OllamaChatGenerator

Generate chat responses using models served with Ollama.

Key Features

• Works with any model available through Ollama, including Llama, Mistral, Qwen, and Gemma.
• Accepts and returns messages in ChatMessage format.
• Supports streaming responses through a configurable callback.
• Supports tool calling for agentic workflows.
• Supports structured output with JSON schema validation.
• Supports reasoning with extended thinking (think parameter).

Configuration

1. Drag the OllamaChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to the Ollama model you want to use (for example, llama3.2).
   2. Set the url to the address of your running Ollama server. The default is http://localhost:11434.
4. Go to the Advanced tab to configure generation_kwargs, timeout, keep_alive, and response_format.

Connections

OllamaChatGenerator receives a list of ChatMessage objects, typically from PromptBuilder or ChatPromptBuilder. It outputs a list of reply ChatMessage objects you can connect to AnswerBuilder or other downstream components.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  OllamaChatGenerator:
    type: haystack_integrations.components.generators.ollama.chat.chat_generator.OllamaChatGenerator
    init_parameters:
      model: llama3.2
      url: http://localhost:11434
      generation_kwargs:
        temperature: 0.7
      timeout: 120

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: user
        content: |
          Answer the following question: {{ question }}

  llm:
    type: haystack_integrations.components.generators.ollama.chat.chat_generator.OllamaChatGenerator
    init_parameters:
      model: llama3.2
      url: http://localhost:11434
      timeout: 120

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
model	str	qwen3:0.6b	The name of the Ollama model to use. Run ollama list to see available models.
url	str	http://localhost:11434	The URL of the Ollama API server.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional parameters for text generation, such as temperature, top_k, top_p, and num_predict.
timeout	int	120	Request timeout in seconds.
max_retries	int	0	Maximum number of retries on failed requests.
keep_alive	Optional[Union[float, str]]	None	Controls how long the model stays loaded in memory after the last request.
streaming_callback	Optional[Callable]	None	A callback function for streaming responses.
tools	Optional[List[Tool]]	None	A list of tools the model can use for tool calling.
response_format	Optional[Union[Literal["json"], Dict]]	None	Forces a specific response format. Set to "json" for JSON output or provide a JSON schema.
think	Union[bool, Literal["low", "medium", "high"]]	False	Enables extended reasoning. Set to True or a verbosity level string for reasoning models.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
generation_kwargs	Optional[Dict[str, Any]]	None	Generation parameters to override init-time values.
tools	Optional[List[Tool]]	None	Tools to make available to the model, overriding init-time tools.

Related Information

• Pipeline Components