FileTypeClassifier classifies files based on their extension and sends them to a file converter that can handle them. For example, if you have a collection of files of different types, such as PDFs, text files, Markdown, and HTML files, you can use FileTypeClassifier to route them to converters that can further process them into Documents.

FileTypeClassifier takes a file path as input and outputs the same path on the output edge that corresponds to the file's extension. Output edge 1 serves text files and output edge 2 serves PDF files.

Note that deepset Cloud currently supports TXT and PDF file types.

Basic Information

• Pipeline type: Used in indexing pipelines.
• Position in a pipeline: At the very beginning of an indexing pipeline.
• Node input: File name
• Node output: File name (routed)
• Available node classes: FileTypeClassifier

Outgoing Edges

By default, the FileTypeClassifier has five outgoing edges. It routes an incoming file through one of these to a file converter, such as PDFToTextConverter or TextConverter, which then converts them into Document.

These are the default outgoing edges:

Usage Example

You can use FileTypeClassifier as the first node in your indexing pipeline. First, define it in the components section of your pipeline definition file and then in the pipelines section:

components:
  - name: FileClassifier
    type: FileTypeClassifier
    params:
      supported_types: ["txt", "pdf"]
    ...
 pipelines:
  - name: indexing
    nodes:
      - name: FileClassifier
        inputs: [File]
        #then you specify the input for the file converters:
      - name: PDFToTextConverter
        inputs: [FileClassifier.output_2] #this is output_2 because output edge 2 serves PDF files
      - name: TextConverter
        inputs: [FileClassifier.output_1] 
#TextConverter takes output_1 because output edge 1 handles TXT files

Arguments

You can specify the following arguments for FileTypeClassifier: