PDFMinerToDocument

Convert PDF files to documents using PDFMiner extraction parameters.

Basic Information

• Type: haystack.components.converters.pdfminer.PDFMinerToDocument
• Components it can connect with:
  • FileTypeRouter: PDFMinerToDocument can receive filtered PDF files from FileTypeRouter.
  • Preprocessors: PDFMinerToDocument can send converted documents to a preprocessor such as DocumentSplitter.

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of PDF file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		Converted documents.

Overview

PDFMinerToDocument uses pdfminer compatible converters to convert PDF files to documents. For details, see PDFMiner documentation.

Usage Example

Initializing the Component

components:
  PDFMinerToDocument:
    type: components.converters.pdfminer.PDFMinerToDocument
    init_parameters:

Using the Component in an Index

This is a simple index you can use if you only have PDF files to index.

components:
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
      skip_empty_documents: true
  DeepsetNvidiaDocumentEmbedder:
    type: deepset_cloud_custom_nodes.embedders.nvidia.document_embedder.DeepsetNvidiaDocumentEmbedder
    init_parameters:
      model: intfloat/multilingual-e5-base
      prefix: ''
      suffix: ''
      batch_size: 32
      meta_fields_to_embed:
      embedding_separator: \n
      truncate:
      normalize_embeddings: true
      timeout:
      backend_kwargs:
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: ''
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
  PDFMinerToDocument:
    type: haystack.components.converters.pdfminer.PDFMinerToDocument
    init_parameters:
      line_overlap: 0.5
      char_margin: 2
      line_margin: 0.5
      word_margin: 0.1
      boxes_flow: 0.5
      detect_vertical: true
      all_texts: false
      store_full_path: false

connections:
- sender: DocumentSplitter.documents
  receiver: DeepsetNvidiaDocumentEmbedder.documents
- sender: DeepsetNvidiaDocumentEmbedder.documents
  receiver: DocumentWriter.documents
- sender: PDFMinerToDocument.documents
  receiver: DocumentSplitter.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - PDFMinerToDocument.sources

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
line_overlap	float	0.5	Determines whether two characters are considered to be on the same line based on the amount of overlap between them. The overlap is calculated relative to the minimum height of both characters.
char_margin	float	2	Determines whether two characters are part of the same line based on the distance between them. If the distance is less than the margin specified, the characters are considered to be on the same line. The margin is calculated relative to the width of the character.
word_margin	float	0.1	Determines whether two characters on the same line are part of the same word based on the distance between them. If the distance is greater than the margin specified, an intermediate space will be added between them to make the text more readable. The margin is calculated relative to the width of the character.
line_margin	float	0.5	Determines whether two lines are part of the same paragraph based on the distance between them. If the distance is less than the margin specified, the lines are considered to be part of the same paragraph. The margin is calculated relative to the height of a line.
boxes_flow	Optional[float]	0.5	Determines the importance of horizontal and vertical position when determining the order of text boxes. A value between -1.0 and +1.0 can be set, with -1.0 indicating that only horizontal position matters and +1.0 indicating that only vertical position matters. Setting the value to 'None' disables advanced layout analysis, and text boxes are ordered based on the position of their bottom left corner.
detect_vertical	bool	True	Determines whether vertical text should be considered during layout analysis.
all_texts	bool	False	If layout analysis should be performed on text in figures.
store_full_path	bool	False	If True, the full path of the file is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of PDF file paths or ByteStream objects.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the Documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.