MultiFileConverter

Convert multiple file types to documents in a single operation. The component automatically detects each file's type and applies the appropriate converter.

MultiFileConverter is a SuperComponent that combines FileTypeRouter, nine converters, and DocumentJoiner into a single component. It automatically detects file types and applies the appropriate converter for each file, so you don't need to manually connect individual converter components.

Key Features

• Automatically detects file types and routes each file to the appropriate converter.
• Supports nine file types: CSV, DOCX, HTML, JSON, Markdown, plain text, PDF, PPTX, and XLSX.
• Returns a list of failed files separately so you can handle conversion errors.
• Simplifies pipeline design by replacing multiple converter components with one.

Configuration

1. Drag the MultiFileConverter component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. Configure the component settings:
   • Set the Encoding for text-based files. The default is UTF-8.
   • Set the JSON Content Key to specify which key to use as document content when converting JSON files. The default is content.

Connections

MultiFileConverter accepts a list of file paths or ByteStream objects through its sources input. It outputs a list of successfully converted Document objects and a list of file paths that failed to convert.

It typically connects with:

• FilesInput: receives files to convert.
• DocumentSplitter or document embedding components: sends converted documents for further processing.

Source Code

To check this component's source code, open multi_file_converter.py in the Haystack repository.

Usage Examples

Basic Configuration

  MultiFileConverter:
    type: haystack.components.converters.multi_file_converter.MultiFileConverter
    init_parameters:
      encoding: utf-8
      json_content_key: content

Using the Component in an Index

In this index, MultiFileConverter receives files, converts them, and then sends them to DocumentSplitter.

# haystack-pipeline
components:
  MultiFileConverter:
    type: haystack.components.converters.multi_file_converter.MultiFileConverter
    init_parameters:
      encoding: utf-8
      json_content_key: content
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
      skip_empty_documents: true
  DeepsetNvidiaDocumentEmbedder:
    type: deepset_cloud_custom_nodes.embedders.nvidia.document_embedder.DeepsetNvidiaDocumentEmbedder
    init_parameters:
      model: intfloat/multilingual-e5-base
      prefix: ''
      suffix: ''
      batch_size: 32
      meta_fields_to_embed:
      embedding_separator: \n
      truncate:
      normalize_embeddings: true
      timeout:
      backend_kwargs:
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: ''
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:

connections:
- sender: MultiFileConverter.documents
  receiver: DocumentSplitter.documents
- sender: DocumentSplitter.documents
  receiver: DeepsetNvidiaDocumentEmbedder.documents
- sender: DeepsetNvidiaDocumentEmbedder.documents
  receiver: DocumentWriter.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - MultiFileConverter.sources

Parameters

Inputs

Parameter	Type	Description
sources	List[Union[str, Path, ByteStream]]	A list of file paths or byte streams to convert.

Outputs

Parameter	Type	Description
documents	List[Document]	A list of converted documents. Only included if at least one file is successfully converted.
failed	List[str]	A list of files that failed to convert.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
encoding	str	utf-8	The encoding to use when reading text files.
json_content_key	str	content	The key to extract content from JSON files.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Description
sources	List[Union[str, Path, ByteStream]]	A list of file paths or byte streams to convert.