WatsonxChatGenerator

Generate chat responses using IBM watsonx.ai foundation models.

Key Features

• Supports IBM's foundation models available on watsonx.ai, including Granite and Llama models.
• Accepts and returns messages in ChatMessage format.
• Supports multimodal inputs with text and images.
• Supports streaming responses through a configurable callback.
• Supports tool calling for agentic workflows.
• Configurable IBM Cloud region through api_base_url.

Configuration

1. Drag the WatsonxChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to the watsonx.ai model you want to use (for example, ibm/granite-4-h-small). For a full list, see the IBM watsonx.ai documentation.
   2. Create secrets for your watsonx.ai credentials. Use WATSONX_API_KEY for the API key and WATSONX_PROJECT_ID for the project ID. For instructions, see Create Secrets.
4. Go to the Advanced tab to configure generation_kwargs, api_base_url, timeout, and tools.

Connections

WatsonxChatGenerator receives a list of ChatMessage objects, typically from PromptBuilder or ChatPromptBuilder. It outputs a list of reply ChatMessage objects you can connect to AnswerBuilder or other downstream components.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  WatsonxChatGenerator:
    type: haystack_integrations.components.generators.watsonx.chat.WatsonxChatGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - WATSONX_API_KEY
        strict: false
      project_id:
        type: env_var
        env_vars:
        - WATSONX_PROJECT_ID
        strict: false
      model: ibm/granite-4-h-small
      api_base_url: https://us-south.ml.cloud.ibm.com
      generation_kwargs:
        max_tokens: 1024

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: user
        content: "Answer the following question: {{ question }}"

  llm:
    type: haystack_integrations.components.generators.watsonx.chat.WatsonxChatGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - WATSONX_API_KEY
        strict: false
      project_id:
        type: env_var
        env_vars:
        - WATSONX_PROJECT_ID
        strict: false
      model: ibm/granite-4-h-small
      generation_kwargs:
        max_tokens: 1024

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var("WATSONX_API_KEY")	The IBM Cloud API key for watsonx.ai.
model	str	ibm/granite-4-h-small	The name of the watsonx.ai foundation model. For a full list, see the IBM documentation.
project_id	Secret	Secret.from_env_var("WATSONX_PROJECT_ID")	The watsonx.ai project ID.
api_base_url	str	https://us-south.ml.cloud.ibm.com	The IBM Cloud watsonx.ai API base URL. Change the region prefix (for example, eu-de) to use a different IBM Cloud region.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters for the watsonx.ai API, such as max_tokens, temperature, top_p, and stop_sequences.
timeout	Optional[float]	None	Request timeout in seconds.
max_retries	Optional[int]	None	Maximum number of retries on API errors.
verify	Optional[Union[bool, str]]	None	SSL certificate verification setting.
streaming_callback	Optional[Callable]	None	A callback function for streaming responses.
tools	Optional[List[Tool]]	None	A list of tools the model can use for tool calling.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
generation_kwargs	Optional[Dict[str, Any]]	None	Generation parameters to override init-time values.
streaming_callback	Optional[Callable]	None	A callback function to override the init-time streaming callback.
tools	Optional[List[Tool]]	None	Tools to make available to the model.

Related Information

• Pipeline Components