TextCleaner

Clean text strings by removing patterns, converting to lowercase, or removing punctuation and numbers.

Key Features

• Removes substrings matching configurable regular expression patterns.
• Converts all characters to lowercase for text normalization.
• Removes punctuation from text strings.
• Removes numerical digits from text strings.
• Works on plain text strings, unlike DocumentCleaner which operates on Document objects.
• Useful for normalizing generated responses before evaluation or comparison.

Configuration

1. Drag the TextCleaner component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. Configure the parameters as needed.

Connections

TextCleaner accepts a list of text strings (texts) as input and outputs a list of cleaned text strings (texts).

Connect components that output text strings to the input — such as generators that produce text replies. Connect the output to evaluation components for cleaned text comparison.

Usage Example

Using the Component in a Pipeline

This example shows a pipeline that cleans generated answers before evaluation.

components:
  retriever:
    type: haystack_integrations.components.retrievers.opensearch.embedding_retriever.OpenSearchEmbeddingRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: ''
          embedding_dim: 384
          return_embedding: false
          create_index: true
          similarity: cosine
      top_k: 5
  text_embedder:
    type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
    init_parameters:
      model: sentence-transformers/all-MiniLM-L6-v2
  prompt_builder:
    type: haystack.components.builders.prompt_builder.PromptBuilder
    init_parameters:
      template: |-
        Answer the question based on the context.
        Context: {% for doc in documents %}{{ doc.content }}{% endfor %}
        Question: {{ question }}
        Answer:
  generator:
    type: haystack.components.generators.openai.OpenAIGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - OPENAI_API_KEY
        strict: true
      model: gpt-4o-mini
  TextCleaner:
    type: haystack.components.preprocessors.text_cleaner.TextCleaner
    init_parameters:
      remove_regexps:
      convert_to_lowercase: true
      remove_punctuation: false
      remove_numbers: false

connections:
- sender: text_embedder.embedding
  receiver: retriever.query_embedding
- sender: retriever.documents
  receiver: prompt_builder.documents
- sender: prompt_builder.prompt
  receiver: generator.prompt
- sender: generator.replies
  receiver: TextCleaner.texts

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - text_embedder.text
  - prompt_builder.question

outputs:
  answers: TextCleaner.texts

Parameters

Inputs

Parameter	Type	Default	Description
texts	List[str]		List of strings to clean.

Outputs

Parameter	Type	Default	Description
texts	List[str]		List of cleaned text strings.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
remove_regexps	Optional[List[str]]	None	A list of regex patterns to remove matching substrings from the text.
convert_to_lowercase	bool	False	If True, converts all characters to lowercase.
remove_punctuation	bool	False	If True, removes punctuation from the text.
remove_numbers	bool	False	If True, removes numerical digits from the text.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
texts	List[str]		List of strings to clean.