TikaDocumentConverter

Convert files of various types to documents you can index into a document store. This component uses Apache Tika for parsing, which supports a wide range of file formats.

Apache Tika supports a wide range of formats:

• Document formats: Ms Office (DOCX, XLSX, PPTX, PPT, RTF), OpenDocument (ODT, ODS, ODP), PDF (with text extraction, no OCR), Markup languages (HTML, XML, XHTML, Markdown), Plain text (TXT, CSV, TSV)
• Archive formats: ZIP, RAR, 7Z, TAR, GZIP, BZIP2
• Email formats: MDG, EML, MBOX, PST, OST

To see the full list, see the Tika documentation.

Limitations

• No OCR support.
• Requires a running Tika server.

Key Features

• Converts a wide range of file formats using Apache Tika.
• Especially useful for archive formats (ZIP, RAR, 7Z) and email formats (EML, MSG) not supported by other converters.
• Configurable Tika server URL.
• Can be combined with DocumentJoiner to merge output from multiple converters.

Configuration

1. Drag the TikaDocumentConverter component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   • Set the tika_url to point to your running Tika server (for example, http://localhost:9998/tika). For more options on running Tika, see the Tika official documentation.
4. Go to the Advanced tab to configure store_full_path.

Connections

TikaDocumentConverter receives files from FileTypeRouter. It sends converted documents to DocumentJoiner, which is useful if you have multiple converters in your pipeline and want to join their output.

Source Code

To check this component's source code, open tika.py in the Haystack repository.

Usage Examples

Basic Configuration

  TikaDocumentConverter:
    type: haystack.components.converters.tika.TikaDocumentConverter
    init_parameters:
      tika_url: http://localhost:9998/tika
      store_full_path: false

1. Drag the TikaDocumentConverter component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. Configure the parameters as needed.

Connections

TikaDocumentConverter accepts a list of file paths or ByteStream objects as input. It outputs a list of converted Haystack documents.

Connect FileTypeRouter to its sources input for multi-format indexing. Connect its documents output to DocumentJoiner to merge with output from other converters, or directly to DocumentSplitter.

Usage Example

In this index, TikaDocumentConverter handles multiple file types including documents, archives, and various formats that other converters might not support.

components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/plain
      - application/pdf
      - text/markdown
      - text/html
      - application/vnd.openxmlformats-officedocument.wordprocessingml.document
      - application/vnd.openxmlformats-officedocument.presentationml.presentation
      - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
      - text/csv
      - application/zip
      - application/x-rar-compressed
      - application/x-7z-compressed
      - message/rfc822
      - application/vnd.ms-outlook
      additional_mimetypes:
        application/vnd.openxmlformats-officedocument.wordprocessingml.document: .docx
        application/vnd.openxmlformats-officedocument.presentationml.presentation: .pptx
        application/vnd.openxmlformats-officedocument.spreadsheetml.sheet: .xlsx
        application/x-rar-compressed: .rar
        application/x-7z-compressed: .7z
        message/rfc822: .eml
        application/vnd.ms-outlook: .msg

  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  PDFMinerToDocument:
    type: haystack.components.converters.pdfminer.PDFMinerToDocument
    init_parameters:
      line_overlap: 0.5
      char_margin: 2
      line_margin: 0.5
      word_margin: 0.1
      boxes_flow: 0.5
      detect_vertical: true
      all_texts: false
      store_full_path: false

  MarkdownToDocument:
    type: haystack.components.converters.markdown.MarkdownToDocument
    init_parameters: {}

  HTMLToDocument:
    type: haystack.components.converters.html.HTMLToDocument
    init_parameters:
      extraction_kwargs:
        output_format: markdown
        target_language:
        include_tables: true
        include_links: true

  DOCXToDocument:
    type: haystack.components.converters.docx.DOCXToDocument
    init_parameters:
      link_format: markdown

  PPTXToDocument:
    type: haystack.components.converters.pptx.PPTXToDocument
    init_parameters:
      store_full_path: false

  XLSXToDocument:
    type: haystack.components.converters.xlsx.XLSXToDocument
    init_parameters:
      store_full_path: false

  CSVToDocument:
    type: haystack.components.converters.csv.CSVToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  TikaDocumentConverter:
    type: haystack.components.converters.tika.TikaDocumentConverter
    init_parameters:
      tika_url: http://localhost:9998/tika
      store_full_path: false

  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters: {}

  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 1000
      split_overlap: 200

  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack.document_stores.opensearch.OpenSearchDocumentStore
        init_parameters:
          host: localhost
          port: 9200
          username: admin
          password:
            type: env_var
            env_vars:
            - OPENSEARCH_PASSWORD
            strict: false
          index: documents
          embedding_dim: 768
          similarity: cosine

connections:
  - sender: FileTypeRouter.text/plain
    receiver: TextFileToDocument.sources
  - sender: FileTypeRouter.application/pdf
    receiver: PDFMinerToDocument.sources
  - sender: FileTypeRouter.text/markdown
    receiver: MarkdownToDocument.sources
  - sender: FileTypeRouter.text/html
    receiver: HTMLToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.wordprocessingml.document
    receiver: DOCXToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.presentationml.presentation
    receiver: PPTXToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    receiver: XLSXToDocument.sources
  - sender: FileTypeRouter.text/csv
    receiver: CSVToDocument.sources
  - sender: FileTypeRouter.application/zip
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/x-rar-compressed
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/x-7z-compressed
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.message/rfc822
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/vnd.ms-outlook
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.unclassified
    receiver: TikaDocumentConverter.sources
  - sender: TextFileToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: PDFMinerToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: MarkdownToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: HTMLToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: DOCXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: PPTXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: XLSXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: CSVToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: TikaDocumentConverter.documents
    receiver: DocumentJoiner.documents
  - sender: DocumentJoiner.documents
    receiver: DocumentSplitter.documents
  - sender: DocumentSplitter.documents
    receiver: DocumentWriter.documents

Parameters

Inputs

Parameter	Type	Description
sources	List[Union[str, Path, ByteStream]]	List of file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.

Outputs

Parameter	Type	Description
documents	List[Document]	Converted documents.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
tika_url	str	http://localhost:9998/tika	Tika server URL.
store_full_path	bool	False	If True, the full path of the file is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.