TikaDocumentConverter

Converts files of different types to documents you can index into a document store.

Basic Information

Type: haystack.components.converters.tika.TikaDocumentConverter
Components it can connect with:
- FileTypeRouter: TikaDocumentConverter can receive files from FileTypeRouter.
- DocumentJoiner: TikaDocumentConverter can send converted documents to DocumentJoiner. This is useful if you have other converters in your pipeline and want to join their output with TikaDocumentConverter's output before sending it further down the pipeline.

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of HTML file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If `sources` contains ByteStream objects, their `meta` will be added to the output Documents.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		Converted documents.

Overview

TikaDocumentConverter uses Apache Tika for parsing the files. Apache Tika is a powerful content detection and extraction library. To learn more, see Apache TikaIt requires a running Tika server. For more options on running Tika, see the Tika official documentation.

Apache Tika supports a wide range of formats:

Document formats:
- Ms Office: DOCX, XLSX, PPTX, PPT, RTF
- OpenDocument: ODT, ODS, ODP
- PDF (with text extraction, no OCR)
- Markup languages: HTML, XML, XHTML, Markdown
- Plain text: TXT, CSV, TSV
Archive formats:
- Compressed files: ZIP, RAR, 7Z, TAR, GZIP, BZIP2
Email formats: MDG, EML, MBOX, PST, OST To see the full list of supported formats, see the Tika documentation.

Limitations

No OCR support.
Requires a running Tika server.

Usage Example

Initializing the Component

components:
  TikaDocumentConverter:
    type: components.converters.tika.TikaDocumentConverter
    init_parameters:

Using the Component in an Index

In this index, TikaDocumentConverter handles multiple file types including documents, archives, and various formats that other converters might not support.

components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/plain
      - application/pdf
      - text/markdown
      - text/html
      - application/vnd.openxmlformats-officedocument.wordprocessingml.document
      - application/vnd.openxmlformats-officedocument.presentationml.presentation
      - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
      - text/csv
      - application/zip
      - application/x-rar-compressed
      - application/x-7z-compressed
      - message/rfc822
      - application/vnd.ms-outlook
      additional_mimetypes:
        application/vnd.openxmlformats-officedocument.wordprocessingml.document: .docx
        application/vnd.openxmlformats-officedocument.presentationml.presentation: .pptx
        application/vnd.openxmlformats-officedocument.spreadsheetml.sheet: .xlsx
        application/x-rar-compressed: .rar
        application/x-7z-compressed: .7z
        message/rfc822: .eml
        application/vnd.ms-outlook: .msg

  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  PDFMinerToDocument:
    type: haystack.components.converters.pdfminer.PDFMinerToDocument
    init_parameters:
      line_overlap: 0.5
      char_margin: 2
      line_margin: 0.5
      word_margin: 0.1
      boxes_flow: 0.5
      detect_vertical: true
      all_texts: false
      store_full_path: false

  MarkdownToDocument:
    type: haystack.components.converters.markdown.MarkdownToDocument
    init_parameters: {}

  HTMLToDocument:
    type: haystack.components.converters.html.HTMLToDocument
    init_parameters:
      extraction_kwargs:
        output_format: markdown
        target_language:
        include_tables: true
        include_links: true

  DOCXToDocument:
    type: haystack.components.converters.docx.DOCXToDocument
    init_parameters:
      link_format: markdown

  PPTXToDocument:
    type: haystack.components.converters.pptx.PPTXToDocument
    init_parameters:
      store_full_path: false

  XLSXToDocument:
    type: haystack.components.converters.xlsx.XLSXToDocument
    init_parameters:
      store_full_path: false

  CSVToDocument:
    type: haystack.components.converters.csv.CSVToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false

  TikaDocumentConverter:
    type: haystack.components.converters.tika.TikaDocumentConverter
    init_parameters:
      tika_url: http://localhost:9998/tika
      store_full_path: false

  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters: {}

  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 1000
      split_overlap: 200

  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack.document_stores.opensearch.OpenSearchDocumentStore
        init_parameters:
          host: localhost
          port: 9200
          username: admin
          password:
            type: env_var
            env_vars:
            - OPENSEARCH_PASSWORD
            strict: false
          index: documents
          embedding_dim: 768
          similarity: cosine

connections:
  - sender: FileTypeRouter.text/plain
    receiver: TextFileToDocument.sources
  - sender: FileTypeRouter.application/pdf
    receiver: PDFMinerToDocument.sources
  - sender: FileTypeRouter.text/markdown
    receiver: MarkdownToDocument.sources
  - sender: FileTypeRouter.text/html
    receiver: HTMLToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.wordprocessingml.document
    receiver: DOCXToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.presentationml.presentation
    receiver: PPTXToDocument.sources
  - sender: FileTypeRouter.application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    receiver: XLSXToDocument.sources
  - sender: FileTypeRouter.text/csv
    receiver: CSVToDocument.sources
  - sender: FileTypeRouter.application/zip
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/x-rar-compressed
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/x-7z-compressed
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.message/rfc822
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.application/vnd.ms-outlook
    receiver: TikaDocumentConverter.sources
  - sender: FileTypeRouter.unclassified
    receiver: TikaDocumentConverter.sources
  - sender: TextFileToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: PDFMinerToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: MarkdownToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: HTMLToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: DOCXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: PPTXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: XLSXToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: CSVToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: TikaDocumentConverter.documents
    receiver: DocumentJoiner.documents
  - sender: DocumentJoiner.documents
    receiver: DocumentSplitter.documents
  - sender: DocumentSplitter.documents
    receiver: DocumentWriter.documents

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
tika_url	str	http://localhost:9998/tika	Tika server URL.
store_full_path	bool	False	If `True`, the full path of the file is stored in the metadata of the document. If `False`, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of HTML file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If `sources` contains ByteStream objects, their `meta` will be added to the output Documents.

Was this page helpful?

Basic Information​

Inputs​

Outputs​

Overview​

Limitations​

Usage Example​

Initializing the Component​

Using the Component in an Index​

Parameters​

Init Parameters​

Run Method Parameters​