TransformersChatGenerator

Generate chat responses using transformer models from Hugging Face that run locally. Use this component when you want to run open-source LLMs on your own hardware without external API calls.

Key Features

• Runs Hugging Face transformers models locally for fully offline operation.
• Supports text-generation, text2text-generation, and image-text-to-text pipeline tasks.
• Supports tool calling with a configurable tool parsing function.
• Supports streaming responses through a configurable callback.
• Supports extended thinking mode (for example, for Qwen3 models).
• Configurable device placement (CPU, GPU, or automatic).

Configuration

1. Drag the TransformersChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to the Hugging Face model you want to use (for example, Qwen/Qwen3-0.6B or meta-llama/Llama-2-7b-chat-hf).
   2. If the model requires authentication, create a secret with your Hugging Face API token and set it as token. Use HF_API_TOKEN or HF_TOKEN as the environment variable name. For instructions, see Create Secrets.
4. Go to the Advanced tab to configure generation_kwargs such as max_new_tokens, temperature, and do_sample.

note

LLMs running locally may need powerful hardware. Ensure your deployment environment has sufficient GPU or CPU resources for the model you choose.

Connections

TransformersChatGenerator receives a list of ChatMessage objects as input, typically from PromptBuilder or ChatPromptBuilder. It outputs a list of reply ChatMessage objects you can connect to AnswerBuilder or other downstream components.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  TransformersChatGenerator:
    type: haystack_integrations.components.generators.transformers.chat.chat_generator.TransformersChatGenerator
    init_parameters:
      model: Qwen/Qwen3-0.6B
      generation_kwargs:
        max_new_tokens: 512
        temperature: 0.7
        do_sample: true

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: system
        content: You are a helpful assistant. Answer questions based on the provided documents.
      - role: user
        content: |
          Documents:
          {% for doc in documents %}
          {{ doc.content }}
          {% endfor %}
          Question: {{ question }}

  llm:
    type: haystack_integrations.components.generators.transformers.chat.chat_generator.TransformersChatGenerator
    init_parameters:
      model: Qwen/Qwen3-0.6B
      token:
        type: env_var
        env_vars:
        - HF_API_TOKEN
        - HF_TOKEN
        strict: false
      generation_kwargs:
        max_new_tokens: 512

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question
  documents:
  - prompt_builder.documents

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
model	str	Qwen/Qwen3-0.6B	The Hugging Face model name or path to use.
task	Optional[str]	None	The pipeline task. One of text-generation, text2text-generation, or image-text-to-text. If None, the task is inferred from the model.
device	Optional[ComponentDevice]	None	The device to run the model on. If None, uses the default device.
token	Optional[Secret]	Secret.from_env_var(["HF_API_TOKEN", "HF_TOKEN"], strict=False)	A Hugging Face API token for accessing gated or private models.
chat_template	Optional[str]	None	A custom Jinja2 chat template string. If None, uses the model's default template.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters such as max_new_tokens, temperature, do_sample, and top_p.
huggingface_pipeline_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments passed directly to the Hugging Face pipeline() constructor.
stop_words	Optional[List[str]]	None	A list of stop words that cause generation to stop early.
streaming_callback	Optional[Callable]	None	A callback function for streaming responses token by token.
tools	Optional[List[Tool]]	None	A list of tools the model can use.
tool_parsing_function	Optional[Callable]	None	A custom function to parse tool call strings from the model output.
enable_thinking	bool	False	Whether to enable extended thinking mode. Supported by models like Qwen3.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters to override init-time values.
streaming_callback	Optional[Callable]	None	A callback function to override the init-time streaming callback.
tools	Optional[List[Tool]]	None	Tools to make available to the model, overriding init-time tools.

Related Information

• Pipeline Components