Cleaners and Splitters
These components are used in indexing pipelines to prepare your data for search by normalizing whitespaces, cleaning empty lines, or splitting documents into smaller chunks.
- DocumentCleaner: Makes document text more readable by removing extra whitespaces, empty lines, and the like.
- DocumentSplitter: Splits documents into shorter chunks.
- TextCleaner: Removes regexes, punctuation, and numbers from text.
Updated about 1 month ago