DocumentCleaner

Clean the text in documents by removing whitespaces, empty lines, headers, footers, and more.

Basic Information

• Type: haystack.components.preprocessors.document_cleaner.DocumentCleaner
• Components it can connect with:
  • Converters: DocumentCleaner receives documents from converters like TextFileToDocument, PDFMinerToDocument, or AzureOCRDocumentConverter.
  • DocumentSplitter: DocumentCleaner typically sends cleaned documents to DocumentSplitter for chunking.
  • Embedders: DocumentCleaner can send documents directly to document embedders if no splitting is needed.

Inputs

Parameter	Type	Default	Description
documents	List[Document]		List of Documents to clean.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		List of cleaned documents.

Overview

DocumentCleaner makes text documents more readable by removing unwanted content. This is useful for preparing documents for further processing by LLMs.

The cleaning steps are executed in the following order:

1. Unicode normalization (if enabled)
2. ASCII conversion (if enabled)
3. Remove extra whitespaces
4. Remove empty lines
5. Remove specified substrings
6. Remove regex matches
7. Remove repeated substrings (headers/footers)

Key parameters:

• remove_empty_lines: Removes empty lines from the document
• remove_extra_whitespaces: Removes extra whitespaces from the document
• remove_repeated_substrings: Removes repeated headers/footers from pages (requires form feed character "\f" as page separator)
• remove_substrings: List of specific strings to remove
• remove_regex: Regular expression pattern to match and remove
• unicode_normalization: Normalizes Unicode characters (NFC, NFKC, NFD, or NFKD)
• ascii_only: Converts text to ASCII only, removing accents

Usage Example

Using the Component in an Index

This example shows a typical indexing pipeline where DocumentCleaner cleans documents after conversion and before splitting.

components:
  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false
  DocumentCleaner:
    type: haystack.components.preprocessors.document_cleaner.DocumentCleaner
    init_parameters:
      remove_empty_lines: true
      remove_extra_whitespaces: true
      remove_repeated_substrings: false
      remove_substrings:
      remove_regex:
      keep_id: false
      unicode_normalization:
      ascii_only: false
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: documents-index
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          create_index: true
          similarity: cosine
      policy: NONE

connections:
- sender: TextFileToDocument.documents
  receiver: DocumentCleaner.documents
- sender: DocumentCleaner.documents
  receiver: DocumentSplitter.documents
- sender: DocumentSplitter.documents
  receiver: DocumentWriter.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - TextFileToDocument.sources

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
remove_empty_lines	bool	True	If True, removes empty lines.
remove_extra_whitespaces	bool	True	If True, removes extra whitespaces.
remove_repeated_substrings	bool	False	If True, removes repeated substrings (headers and footers) from pages. Pages must be separated by a form feed character "\f", which is supported by TextFileToDocument and AzureOCRDocumentConverter.
remove_substrings	Optional[List[str]]	None	List of substrings to remove from the text.
remove_regex	Optional[str]	None	Regex to match and replace substrings by "".
keep_id	bool	False	If True, keeps the IDs of the original documents.
unicode_normalization	Optional[Literal['NFC', 'NFKC', 'NFD', 'NFKD']]	None	Unicode normalization form to apply to the text. Note: This will run before any other steps.
ascii_only	bool	False	Whether to convert the text to ASCII only. Will remove accents from characters and replace them with ASCII characters. Other non-ASCII characters will be removed. Note: This will run before any pattern matching or removal.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
documents	List[Document]		List of Documents to clean.