TextFileToDocument

Convert text files to documents your pipeline can query. Use this component in indexes to write text file contents into a document store.

TextFileToDocument uses UTF-8 encoding by default, but you can set a custom encoding. You can also attach metadata to the resulting documents.

Key Features

• Converts plain text files to documents for indexing and search.
• Configurable encoding with UTF-8 as default.
• Optional metadata attachment to resulting documents.

Configuration

1. Drag the TextFileToDocument component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. Configure the component settings:
   • Set the Encoding for the text files. The default is UTF-8. If encoding is specified in the metadata of a source ByteStream, it overrides this value.
   • Set Store Full Path to control whether the full file path or just the file name is stored in document metadata.

Connections

TextFileToDocument accepts a list of file paths or ByteStream objects through its sources input. It outputs a list of Document objects.

It typically connects with:

• FileTypeRouter: receives text files routed by MIME type.
• DocumentJoiner: sends converted documents to join with output from other converters before further processing.

Source Code

To check this component's source code, open txt.py in the Haystack repository.

Usage Examples

Basic Configuration

  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

Using the Component in an Index

In this index, TextFileToDocument receives text files from FileTypeRouter and sends them to DocumentJoiner.

components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/plain
      - text/csv
      - application/json
      additional_mimetypes:
  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false
  CSVToDocument:
    type: haystack.components.converters.csv.CSVToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false
  JSONConverter:
    type: haystack.components.converters.json.JSONConverter
    init_parameters:
      jq_schema:
      content_key:
      extra_meta_fields:
      store_full_path: false
  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate
      weights:
      top_k:
      sort_by_score: true
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: text-index
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:

connections:
- sender: FileTypeRouter.text/plain
  receiver: TextFileToDocument.sources
- sender: FileTypeRouter.text/csv
  receiver: CSVToDocument.sources
- sender: FileTypeRouter.application/json
  receiver: JSONConverter.sources
- sender: TextFileToDocument.documents
  receiver: DocumentJoiner.documents
- sender: CSVToDocument.documents
  receiver: DocumentJoiner.documents
- sender: JSONConverter.documents
  receiver: DocumentJoiner.documents
- sender: DocumentJoiner.documents
  receiver: DocumentSplitter.documents
- sender: DocumentSplitter.documents
  receiver: DocumentWriter.documents

max_runs_per_component: 100

Parameters

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of text file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced documents. If it's a list, its length must match the number of sources as they're zipped together. For ByteStream objects, their meta is added to the output documents.

Outputs

Parameter	Type	Description
documents	List[Document]	A list of converted documents.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
encoding	str	utf-8	The encoding of the text files to convert. If the encoding is specified in the metadata of a source ByteStream, it overrides this value.
store_full_path	bool	False	If True, the full path of the file is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of text file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced documents. If it's a list, its length must match the number of sources as they're zipped together. For ByteStream objects, their meta is added to the output documents.