CSVToDocument

Convert CSV files to documents your pipeline can query.

Basic Information

• Type: haystack.components.converters.csv.CSVToDocument
• Components it can connect with:
  • FileTypeRouter: CSVToDocument can receive CSV files from FileTypeRouter.
  • DocumentJoiner: CSVToDocument can send converted documents to DocumentJoiner. This is useful if you have other converters in your pipeline and want to join their output with CSVToDocument's output before sending it further down the pipeline.

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of file paths or ByteStream objects to convert..
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced documents. If it's a list, the length of the list must match the number of sources, because the two lists are zipped. If sources contains ByteStream objects, their meta is added to the output documents.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		Created documents.

Overview

Use CSVToDocument to convert CSV files to documents your pipeline can query. By default, the component uses UTF-8 encoding when converting files, but you can also set a custom encoding. You can attach metadata to each resulting document using the meta parameter.

Usage Example

Initializing the Component

components:
  CSVToDocument:
    type: haystack.components.converters.csv.CSVToDocument
    init_parameters:

Using the Component in an Index

In this index, CSVToDocument receives CSV files from FileTypeRouter and sends them to DocumentJoiner.

components:
  CSVToDocument:
    type: haystack.components.converters.csv.CSVToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/csv
      - application/pdf
      additional_mimetypes:
      raise_on_failure: false

  PyPDFToDocument:
    type: haystack.components.converters.pypdf.PyPDFToDocument
    init_parameters:
      extraction_mode: plain
      plain_mode_orientations:
      - 0
      - 90
      - 180
      - 270
      plain_mode_space_width: 200
      layout_mode_space_vertically: true
      layout_mode_scale_weight: 1.25
      layout_mode_strip_rotated: true
      layout_mode_font_height_weight: 1
      store_full_path: false
  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate
      weights:
      top_k:
      sort_by_score: true
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
      skip_empty_documents: true
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: Standard-Index-English
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:

connections:
- sender: FileTypeRouter.text/csv
  receiver: CSVToDocument.sources
- sender: FileTypeRouter.application/pdf
  receiver: PyPDFToDocument.sources
- sender: CSVToDocument.documents
  receiver: DocumentJoiner.documents
- sender: PyPDFToDocument.documents
  receiver: DocumentJoiner.documents
- sender: DocumentJoiner.documents
  receiver: DocumentSplitter.documents
- sender: DocumentSplitter.documents
  receiver: DocumentWriter.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - FileTypeRouter.sources

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
encoding	str	utf-8	The encoding of the CSV files to convert. If the encoding is specified in the metadata of a source ByteStream, it overrides this value.
store_full_path	bool	False	If True, the full file path is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of file paths or ByteStream objects.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output documents.