Pipelines search through documents stored in the document store. Documents are passages of plain text. Use converters to transform files into searchable documents.

If you add a converter to your indexing pipeline, the conversion only happens once when you deploy the pipeline. Your files are not converted every time you run a search.

Basic Information

• Pipeline type: Converters are used in indexing pipelines.
• Position in a pipeline: Either at the very beginning or after a FileTypeClassifier.
• Nodes that can precede converters in a pipeline: FileTypeClassifier
• Nodes that can follow converters in a pipeline: PreProcessor
• Node input: File paths
• Node output: Documents
• Supported types:
  • CNAzureConverter
  • DocxToTextConverter
  • MarkdownConverter
  • PDFToTextConverter
  • PptxConverter
  • TextConverter

Converters Overview

CNAzureConverter

CNAzureConverter extracts text and tables from files and converts them into documents you can store in the document store and use in your pipelines. It uses the Form Recognizer service by Microsoft Azure. It can extract content from the following file types:

• PDF
• JPEG
• PNG
• MBP
• TIFF

You must have an active Azure account and a Form Recognizer or Cognitive Services resource. For information on how to set it up, see Microsoft Azure documentation.

📘

For PDF files, the extracted text is not available in the PDF view in deepset Cloud. So when you search with your pipeline and you choose to View File under an answer, you're not going to see the extracted text in the PDF file that opens. This is because this node is used in the indexing pipeline, which stores the contents of the files in the document store which the query pipeline then uses to search.

DocxToTextConverter

Extracts text from DOCX files.

MarkdownConverter

MarkdownConverter converts Markdown files into plain text documents, removing all structured information, like bullet point lists or code block formatting.

📘

Preprocessing Markdown Files for RAG

If you use an LLM in the query pipeline, TextConverter is more effective for preprocessing Markdown files than MarkdownConverter. LLMs are particularly adept at understanding Markdown file structures that TextConverter retains, that's why we recomment using TextConverter for processing these files.

PDFToTextConverter

The PDFToTextConverter is a fast and lightweight PDF converter that converts PDF files to plain text. It works well with most digitally created or searchable PDFs containing a text layer. It can also work with image-based PDFs (for example, scanned documents).

This converter doesn't extract tables as separate documents of type 'table' but treats them as plain text. You can discard numerical tables by setting the remove_numeric_tables parameter to False.

PptxConverter

This converter extracts text from PPTX files. As PPTX doesn't contain page information, PptxConverter returns a list of texts from each slide in the file.

TextConverter

TextConverter converts plain text files to document objects that pipelines can use for search.

In RAG pipelines, we recommend using TextConverter to preprocess Markdown. LLMs are good at understanding structural information in Markdown files, which can help generate the right answer. Unlike MarkdownConverter, TextConverter preserves this information.

Usage Examples

...
components:
  - name: AzureConverter
    type: CNAzureConverter
    params: 
      endpoint: <Form Recognizer or Cognitive Services endpoint>
      credential_key: <FormRecognizer or Cognitive Services key>
      model_id: prebuilt-read
...

pipelines:
# here comes the query pipeline which we skipped in this example
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        inputs: [File]
      - name: AzureConverter
        inputs: [FileTypeClassifier.output_2] # output_2 is where PDF files are routed
      - name: Preprocessor
        inputs: [AzureConverter]
...

...
components:
  - name: DOCXConverter
    type: DocxConverter
...

pipelines:
# here comes the query pipeline which we skipped in this example
  - name: indexing
    nodes:
      - name: FileClassifier
        inputs: [File] 
      - name: DOCXConverter
        inputs: [FileClassifier.output_4]
      - name: PreProcessor
        inputs: [DOCXConverter]
...

components:
  - name: MarkdownConverter
    type: MarkdownConverter
     params: 
      remove_code_snippets: False 
    ...
pipelines:
# here comes the query pipeline which we skipped in this example
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        inputs: [File]
      - name: MarkdownConverter
        inputs: [FileTypeClassifier.output_3] # output_3 is where Markdown files are routed
      - name: Preprocessor
        inputs: [MarkdownConverter]
...

...
components:
  - name: PDFConverter
    type: PDFToTextConverter
    params: 
    	remove_numeric_tables: True
...

pipelines:
# here comes the query pipeline which we skipped in this example
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        inputs: [File]
      - name: PDFConverter
        inputs: [FileTypeClassifier.output_2] # output_2 is where PDF files are routed
      - name: Preprocessor
        inputs: [PDFConverter]
...

...
components:
  - name: PPTXConverter
    type: PptxConverter
  
...

pipelines:
# here comes the query pipeline which we skipped in this example
  - name: indexing
    nodes:
      - name: PPTXConverter
        inputs: [File] 
      - name: Preprocessor
        inputs: [PPTXConverter]
...

...
components:
  - name: TextFileConverter
    type: TextConverter
...
pipelines:
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        type: FileTypeClassifier
      - name: TextFileConverter
        inputs: [FileTypeClassifier.output_1] # This is where text is routed
      - name: Preprocessor
        inputs: [TextFileConverter]
        
# To use TextConverter for preprocessing Markdown files in pipelines 
# containing FileTypeClassifier, add output_3 as the input for TextConverter, like this:
pipelines:
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        type: FileTypeClassifier
      - name: TextFileConverter
        inputs: [FileTypeClassifier.output_1, FileTypeClassifier.output_3] # Output_3 is where MD files are routed
      - name: Preprocessor
        inputs: [TextFileConverter]

Parameters

CNAzureConverter Parameters

DocxConverter Parameters

MarkdownConverter Parameters

PDFToTextConverter Parameters

PptxConverter Parameters

TextConverter Parameters

There are no parameters for TextConverter that you can configure.