OpenDataLoaderConverterBeta

Beta

This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Convert PDF files to documents you can index into a document store. This component uses the OpenDataLoader PDF library to parse PDFs and produce structured output in markdown, plain text, JSON, or HTML.

Requirements

OpenDataLoaderConverter needs Java 11 or newer available on the system PATH. It only accepts PDF files. File paths must use a .pdf extension. If a ByteStream has a MIME type, it must be application/pdf.

Key Features

• Converts PDF files or ByteStream objects into Haystack documents. Non-PDF sources are rejected.
• Lets you choose the output format: markdown, plain text, JSON, or HTML. The default is markdown.
• Turns image extraction off by default. You can override this through convert_kwargs.
• Passes extra keyword arguments to the underlying OpenDataLoader conversion call.
• Preserves metadata from source ByteStream objects, merges optional user metadata, and adds file_path and output_format to each document.

Configuration

1. Drag the OpenDataLoaderConverter component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. On the General tab, select the Output Format you want OpenDataLoader to produce: markdown, text, json, or html. The default is markdown.
4. Go to the Advanced tab to pass extra conversion arguments through convert_kwargs if needed. Use this for options such as password for encrypted PDFs, table_method, pages, or sanitize. For the full list of supported options, see the OpenDataLoader PDF convert options.

Connections

OpenDataLoaderConverter accepts a list of PDF file paths or ByteStream objects as input. It outputs a list of converted Haystack documents.

Connect FileTypeRouter to its sources input for multi-format indexing pipelines. Connect its documents output to DocumentJoiner to merge with output from other converters, or directly to DocumentSplitter.

Source Code

To check this component's source code, open converter.py in the Haystack Core Integrations repository.

Usage Example

Basic Component Configuration

  OpenDataLoaderConverter:
    type: haystack_integrations.components.converters.opendataloader_pdf.converter.OpenDataLoaderConverter
    init_parameters:
      output_format: markdown
      convert_kwargs:
        table_method: cluster

Using the Component in a Pipeline

In this indexing pipeline, OpenDataLoaderConverter converts uploaded PDF files into markdown-formatted documents before they're split and written to a document store.

# haystack-pipeline
components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - application/pdf

  OpenDataLoaderConverter:
    type: haystack_integrations.components.converters.opendataloader_pdf.converter.OpenDataLoaderConverter
    init_parameters:
      output_format: markdown

  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 1000
      split_overlap: 200

  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          host: localhost
          port: 9200
          username: admin
          password:
            type: env_var
            env_vars:
            - OPENSEARCH_PASSWORD
            strict: false
          index: documents
          embedding_dim: 768
          similarity: cosine

connections:
  - sender: FileTypeRouter.application/pdf
    receiver: OpenDataLoaderConverter.sources
  - sender: OpenDataLoaderConverter.documents
    receiver: DocumentSplitter.documents
  - sender: DocumentSplitter.documents
    receiver: DocumentWriter.documents

Parameters

Inputs

Parameter	Type	Description
sources	List[Union[str, Path, ByteStream]]	PDF file paths or Haystack ByteStream objects to convert. File paths must end with .pdf. If a ByteStream has a MIME type, it must be application/pdf. An empty list returns no documents.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	Optional metadata attached to the generated documents. A single dictionary is applied to every source. A list must contain one dictionary per source. ByteStream metadata is preserved. User-provided metadata overwrites matching keys from the source.

Outputs

Parameter	Type	Description
documents	List[Document]	Converted documents. Each document's content is the extracted text in the selected output format. Document metadata includes file_path and output_format.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
output_format	Literal['markdown', 'text', 'json', 'html']	markdown	Format OpenDataLoader writes into each document's content. The chosen value is also stored in document metadata as output_format.
convert_kwargs	Optional[Dict[str, Any]]	None	Extra arguments passed to opendataloader_pdf.convert. The component sets image_output to off unless you override it here. See Common Convert Options and the OpenDataLoader PDF convert options.

Common Convert Options

These options go in convert_kwargs and are passed through to OpenDataLoader:

Option	Type	Default	Description
password	str		Password for encrypted PDF files.
table_method	str	default	Table detection method: default (border-based) or cluster (border and cluster).
reading_order	str	xycut	Reading order algorithm: off or xycut.
pages	str		Pages to extract, for example 1,3,5-7. All pages if omitted.
include_header_footer	bool	False	Include page headers and footers in the output.
sanitize	bool	False	Replace emails, phone numbers, IP addresses, credit cards, and URLs with placeholders.
keep_line_breaks	bool	False	Keep original line breaks in extracted text.
markdown_page_separator	str		Separator between pages in markdown output. Use %page-number% for page numbers.
use_struct_tree	bool	False	Use the PDF structure tree for reading order when the PDF is tagged.
image_output	str	off	Image output mode. The component sets this to off. Other values are embedded (Base64 data URIs) and external (file references).

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		PDF file paths or Haystack ByteStream objects to convert. File paths must end with .pdf. If a ByteStream has a MIME type, it must be application/pdf.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata attached to the generated documents. A single dictionary is applied to every source. A list must contain one dictionary per source. ByteStream metadata is also preserved.

Related Information

• Available Integrations
• FileTypeRouter
• DocumentJoiner
• DocumentSplitter
• OpenDataLoader PDF convert options