FileTypeClassifier classifies files based on their extension and sends them to a file converter that can handle them. For example, if you have a collection of files of different types, such as PDFs, text files, markdown, and HTML files, you can use FileTypeClassifier to route them to converters that can further process them into documents.

FileTypeClassifier takes a path as input and outputs the same path on the output edge that corresponds to the file's extension.

Usage

You can use FileTypeClassifier as the first node in your indexing pipeline. First, define it in the components section of your pipeline definition file:

components:
  - name: FileClassifier
    type: FileTypeClassifier
    params:
      supported_types: ["txt", "pdf"]

Then, add FileTypeClassifier to your indexing pipeline:

pipelines:
  - name: indexing
    nodes:
      - name: FileClassifier
        inputs: [File]
        #then you specify the input for the file converters:
      - name: PDFToTextConverter
        inputs: [FileClassifier.output_2] #this is output_2 because "PDF" is specified as the second extension in "supported_types"
      - name: TextConverter
        inputs: [FileClassifier.output_1] 
#TextConverter takes output_1 because "txt" is specified 
# as the first extension in "supported_types"

Arguments

You can specify the following arguments for FileTypeClassifier: