PDFMinerToDocument

Convert PDF files to documents using PDFMiner's extraction parameters. Use this component when you need fine-grained control over layout analysis and text extraction from PDFs.

PDFMinerToDocument uses pdfminer-compatible converters to extract text from PDF files. It exposes detailed layout analysis parameters for fine-tuned control over text extraction. For details, see the PDFMiner documentation.

Key Features

• Fine-grained control over PDF text extraction with layout analysis parameters.
• Configurable margins for character, word, and line grouping.
• Optional vertical text detection.
• Optional metadata attachment to resulting documents.

Configuration

1. Drag the PDFMinerToDocument component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. Configure the component settings:
   • This component works with default settings for most PDFs. Adjust the settings below if you need to tune extraction for specific PDF layouts.
   • Set Line Overlap to adjust how characters are grouped onto the same line (default: 0.5).
   • Set Char Margin to control the distance threshold for grouping characters on the same line (default: 2).
   • Set Word Margin to control when an intermediate space is inserted between characters (default: 0.1).
   • Set Line Margin to control the distance threshold for grouping lines into paragraphs (default: 0.5).
   • Set Boxes Flow to control whether horizontal or vertical position takes precedence when ordering text boxes (default: 0.5, range: -1.0 to 1.0).
   • Enable Detect Vertical to include vertical text in layout analysis.
   • Enable All Texts to perform layout analysis on text in figures.
   • Set Store Full Path to control whether the full file path or just the file name is stored in document metadata.

Connections

PDFMinerToDocument accepts a list of file paths or ByteStream objects through its sources input. It outputs a list of Document objects.

It typically connects with:

• FileTypeRouter: receives PDF files routed by MIME type.
• DocumentSplitter or other preprocessors: sends converted documents for further processing.

Source Code

To check this component's source code, open pdfminer.py in the Haystack repository.

Usage Examples

Basic Configuration

  PDFMinerToDocument:
    type: haystack.components.converters.pdfminer.PDFMinerToDocument
    init_parameters:
      line_overlap: 0.5
      char_margin: 2
      line_margin: 0.5
      word_margin: 0.1
      boxes_flow: 0.5
      detect_vertical: true
      all_texts: false
      store_full_path: false

Using the Component in an Index

This is a simple index you can use if you only have PDF files to index.

# haystack-pipeline
components:
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
      skip_empty_documents: true
  DeepsetNvidiaDocumentEmbedder:
    type: deepset_cloud_custom_nodes.embedders.nvidia.document_embedder.DeepsetNvidiaDocumentEmbedder
    init_parameters:
      model: intfloat/multilingual-e5-base
      prefix: ''
      suffix: ''
      batch_size: 32
      meta_fields_to_embed:
      embedding_separator: \n
      truncate:
      normalize_embeddings: true
      timeout:
      backend_kwargs:
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: ''
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
  PDFMinerToDocument:
    type: haystack.components.converters.pdfminer.PDFMinerToDocument
    init_parameters:
      line_overlap: 0.5
      char_margin: 2
      line_margin: 0.5
      word_margin: 0.1
      boxes_flow: 0.5
      detect_vertical: true
      all_texts: false
      store_full_path: false

connections:
- sender: DocumentSplitter.documents
  receiver: DeepsetNvidiaDocumentEmbedder.documents
- sender: DeepsetNvidiaDocumentEmbedder.documents
  receiver: DocumentWriter.documents
- sender: PDFMinerToDocument.documents
  receiver: DocumentSplitter.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - PDFMinerToDocument.sources

Parameters

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of PDF file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.

Outputs

Parameter	Type	Description
documents	List[Document]	Converted documents.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
line_overlap	float	0.5	Determines whether two characters are considered to be on the same line based on the amount of overlap between them. The overlap is calculated relative to the minimum height of both characters.
char_margin	float	2	Determines whether two characters are part of the same line based on the distance between them. If the distance is less than the margin specified, the characters are considered to be on the same line. The margin is calculated relative to the width of the character.
word_margin	float	0.1	Determines whether two characters on the same line are part of the same word based on the distance between them. If the distance is greater than the margin specified, an intermediate space will be added between them to make the text more readable. The margin is calculated relative to the width of the character.
line_margin	float	0.5	Determines whether two lines are part of the same paragraph based on the distance between them. If the distance is less than the margin specified, the lines are considered to be part of the same paragraph. The margin is calculated relative to the height of a line.
boxes_flow	Optional[float]	0.5	Determines the importance of horizontal and vertical position when determining the order of text boxes. A value between -1.0 and +1.0 can be set, with -1.0 indicating that only horizontal position matters and +1.0 indicating that only vertical position matters. Setting the value to 'None' disables advanced layout analysis, and text boxes are ordered based on the position of their bottom left corner.
detect_vertical	bool	True	Determines whether vertical text should be considered during layout analysis.
all_texts	bool	False	If layout analysis should be performed on text in figures.
store_full_path	bool	False	If True, the full path of the file is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of PDF file paths or ByteStream objects.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the Documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.