TikaDocumentConverter

Converts files of various formats to Haystack documents using Apache Tika, which supports a wide range of document, archive, and email formats.

Key Features

Supports a broad range of file formats through Apache Tika, including MS Office, PDF, OpenDocument, archives, and email formats.
Handles file formats that other converters don't support, such as ZIP, RAR, 7Z, and EML.
Works with FileTypeRouter to process files in multi-format indexing pipelines.
Connects to a running Tika server via a configurable URL.
Optional full file path storage in document metadata.

Apache Tika supports:

Document formats: MS Office (DOCX, XLSX, PPTX, PPT, RTF), OpenDocument (ODT, ODS, ODP), PDF (with text extraction, no OCR), markup languages (HTML, XML, XHTML, Markdown), plain text (TXT, CSV, TSV)
Archive formats: ZIP, RAR, 7Z, TAR, GZIP, BZIP2
Email formats: MDG, EML, MBOX, PST, OST

For the full list of supported formats, see the Tika documentation.

Limitations

No OCR support.
Requires a running Tika server.

Configuration

Drag the TikaDocumentConverter component onto the canvas from the Component Library.
Click the component to open the configuration panel.
Configure the parameters as needed.

Connections

TikaDocumentConverter accepts a list of file paths or ByteStream objects as input. It outputs a list of converted Haystack documents.

Connect FileTypeRouter to its sources input for multi-format indexing. Connect its documents output to DocumentJoiner to merge with output from other converters, or directly to DocumentSplitter.

Usage Example

In this index, TikaDocumentConverter handles multiple file types including documents, archives, and various formats that other converters might not support.

components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/plain
      - application/pdf
      - text/markdown
      - text/html
      - application/vnd.openxmlformats-officedocument.wordprocessingml.document
      - application/vnd.openxmlformats-officedocument.presentationml.presentation
      - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
      - text/csv
      - application/zip
      - application/x-rar-compressed
      - application/x-7z-compressed
      - message/rfc822
      - application/vnd.ms-outlook
      additional_mimetypes:
        application/vnd.openxmlformats-officedocument.wordprocessingml.document: .docx
        application/vnd.openxmlformats-officedocument.presentationml.presentation: .pptx
        application/vnd.openxmlformats-officedocument.spreadsheetml.sheet: .xlsx
        application/x-rar-compressed: .rar
        application/x-7z-compressed: .7z
        message/rfc822: .eml
        application/vnd.ms-outlook: .msg

  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  PDFMinerToDocument:
    type: haystack.components.converters.pdfminer.PDFMinerToDocument
    init_parameters:
      line_overlap: 0.5
      char_margin: 2
      line_margin: 0.5
      word_margin: 0.1
      boxes_flow: 0.5
      detect_vertical: true
      all_texts: false
      store_full_path: false

  MarkdownToDocument:
    type: haystack.components.converters.markdown.MarkdownToDocument
    init_parameters: {}

  HTMLToDocument:
    type: haystack.components.converters.html.HTMLToDocument
    init_parameters:
      extraction_kwargs:
        output_format: markdown
        target_language:
        include_tables: true
        include_links: true

  DOCXToDocument:
    type: haystack.components.converters.docx.DOCXToDocument
    init_parameters:
      link_format: markdown

  PPTXToDocument:
    type: haystack.components.converters.pptx.PPTXToDocument
    init_parameters:
      store_full_path: false

  XLSXToDocument:
    type: haystack.components.converters.xlsx.XLSXToDocument
    init_parameters:
      store_full_path: false

  CSVToDocument:
    type: haystack.components.converters.csv.CSVToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  TikaDocumentConverter:
    type: haystack.components.converters.tika.TikaDocumentConverter
    init_parameters:
      tika_url: http://localhost:9998/tika
      store_full_path: false

  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters: {}

  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 1000
      split_overlap: 200

  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack.document_stores.opensearch.OpenSearchDocumentStore
        init_parameters:
          host: localhost
          port: 9200
          username: admin
          password:
            type: env_var
            env_vars:
            - OPENSEARCH_PASSWORD
            strict: false
          index: documents
          embedding_dim: 768
          similarity: cosine

connections:
  - sender: FileTypeRouter.text/plain
    receiver: TextFileToDocument.sources
  - sender: FileTypeRouter.application/pdf
    receiver: PDFMinerToDocument.sources
  - sender: FileTypeRouter.text/markdown
    receiver: MarkdownToDocument.sources
  - sender: FileTypeRouter.text/html
    receiver: HTMLToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.wordprocessingml.document
    receiver: DOCXToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.presentationml.presentation
    receiver: PPTXToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    receiver: XLSXToDocument.sources
  - sender: FileTypeRouter.text/csv
    receiver: CSVToDocument.sources
  - sender: FileTypeRouter.application/zip
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/x-rar-compressed
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/x-7z-compressed
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.message/rfc822
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/vnd.ms-outlook
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.unclassified
    receiver: TikaDocumentConverter.sources
  - sender: TextFileToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: PDFMinerToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: MarkdownToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: HTMLToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: DOCXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: PPTXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: XLSXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: CSVToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: TikaDocumentConverter.documents
    receiver: DocumentJoiner.documents
  - sender: DocumentJoiner.documents
    receiver: DocumentSplitter.documents
  - sender: DocumentSplitter.documents
    receiver: DocumentWriter.documents

Parameters

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of HTML file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If `sources` contains ByteStream objects, their `meta` will be added to the output Documents.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		Converted documents.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
tika_url	str	http://localhost:9998/tika	Tika server URL.
store_full_path	bool	False	If `True`, the full path of the file is stored in the metadata of the document. If `False`, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of HTML file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If `sources` contains ByteStream objects, their `meta` will be added to the output Documents.

Was this page helpful?

Key Features​

Limitations​

Configuration​

Connections​

Usage Example​

Parameters​

Inputs​

Outputs​

Init Parameters​

Run Method Parameters​