FileTypeClassifier classifies files based on their extension and sends them to a file converter that can handle them. For example, if you have a collection of files of different types, such as PDFs, text files, Markdown, and HTML files, you can use FileTypeClassifier to route them to converters that can further process them into Documents.
FileTypeClassifier takes a file path as input and outputs the same path on the output edge that corresponds to the file's extension. Output edge 1 serves text files and output edge 2 serves PDF files.
Note that deepset Cloud currently supports TXT and PDF file types.
- Pipeline type: Used in indexing pipelines.
- Position in a pipeline: At the very beginning of an indexing pipeline.
- Node input: File name
- Node output: File name (routed)
- Available node classes: FileTypeClassifier
By default, the FileTypeClassifier has five outgoing edges. It routes an incoming file through one of these to a file converter, such as PDFToTextConverter or TextConverter, which then converts them into Document.
These are the default outgoing edges:
|Outgoing Edge||File Type|
You can use FileTypeClassifier as the first node in your indexing pipeline. First, define it in the components section of your pipeline definition file and then in the pipelines section:
components: - name: FileClassifier type: FileTypeClassifier params: supported_types: ["txt", "pdf"] ... pipelines: - name: indexing nodes: - name: FileClassifier inputs: [File] #then you specify the input for the file converters: - name: PDFToTextConverter inputs: [FileClassifier.output_2] #this is output_2 because output edge 2 serves PDF files - name: TextConverter inputs: [FileClassifier.output_1] #TextConverter takes output_1 because output edge 1 handles TXT files
You can specify the following arguments for
|A list of strings||File extensions, such as |
|Specifies the file types that this node can distinguish. It's limited to a maximum of 10 file extensions. Lists containing more than 10 items and lists with duplicate elements are not allowed.|
Updated 20 days ago