Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

OllamaChatGenerator

Generate text using models running on Ollama.

Key Features​

  • Connects to models served by Ollama, a project for running LLMs locally.
  • Uses the quantized GGUF format by default, enabling LLMs on standard machines without GPUs.
  • Accepts ChatMessage objects as input and returns generated replies as ChatMessage objects.
  • Supports streaming responses via a callback function.
  • Supports tool calls for agentic workflows.
  • Supports structured JSON output via the response_format parameter.

Configuration​

  1. Drag the OllamaChatGenerator component onto the canvas from the Component Library.
  2. Click on the component to open the configuration panel.
  3. On the General tab:
    • Set the model name. The model must already be available in your running Ollama instance. For a full list of supported models, see Ollama's documentation.
    • Set the url to point to your Ollama server (default: http://localhost:11434).
  4. Go to the Advanced tab to configure timeout, keep_alive, generation_kwargs, and response_format.

Connections​

OllamaChatGenerator accepts a list of ChatMessage objects as input. Connect its messages input to the prompt output of ChatPromptBuilder.

It outputs replies as a list of ChatMessage objects. Connect its replies output through OutputAdapter to DeepsetAnswerBuilder.

Source Code​

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples​

Basic Configuration​

  OllamaChatGenerator:
    type: haystack_integrations.components.generators.ollama.chat.chat_generator.OllamaChatGenerator
    init_parameters:
      model: orca-mini
      url: http://localhost:11434
      timeout: 120

This is an example RAG pipeline with OllamaChatGenerator and DeepsetAnswerBuilder connected through OutputAdapter:

components:
  bm25_retriever: # Selects the most similar documents from the document store
    type: haystack_integrations.components.retrievers.opensearch.bm25_retriever.OpenSearchBM25Retriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: 'Standard-Index-English'
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
      top_k: 20 # The number of results to return
      fuzziness: 0

  query_embedder:
    type: deepset_cloud_custom_nodes.embedders.nvidia.text_embedder.DeepsetNvidiaTextEmbedder
    init_parameters:
      normalize_embeddings: true
      model: intfloat/e5-base-v2

  embedding_retriever: # Selects the most similar documents from the document store
    type: haystack_integrations.components.retrievers.opensearch.embedding_retriever.OpenSearchEmbeddingRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: 'Standard-Index-English'
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
      top_k: 20 # The number of results to return

  document_joiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate

  ranker:
    type: deepset_cloud_custom_nodes.rankers.nvidia.ranker.DeepsetNvidiaRanker
    init_parameters:
      model: intfloat/simlm-msmarco-reranker
      top_k: 8

  meta_field_grouping_ranker:
    type: haystack.components.rankers.meta_field_grouping_ranker.MetaFieldGroupingRanker
    init_parameters:
      group_by: file_id
      subgroup_by:
      sort_docs_by: split_id

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

  ChatPromptBuilder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      template:
      - _content:
        - text: "You are a helpful assistant answering the user's questions based on the provided documents.\nIf the answer is not in the documents, rely on the web_search tool to find information.\nDo not use your own knowledge.\n"
        _role: system
      - _content:
        - text: "Provided documents:\n{% for document in documents %}\nDocument [{{ loop.index }}] :\n{{ document.content }}\n{% endfor %}\n\nQuestion: {{ query }}\n"
        _role: user
      required_variables:
      variables:
  OutputAdapter:
    type: haystack.components.converters.output_adapter.OutputAdapter
    init_parameters:
      template: '{{ replies[0] }}'
      output_type: List[str]
      custom_filters:
      unsafe: false

  OllamaChatGenerator:
    type: haystack_integrations.components.generators.ollama.chat.chat_generator.OllamaChatGenerator
    init_parameters:
      model: orca-mini
      url: http://localhost:11434
      generation_kwargs:
      timeout: 120
      keep_alive:
      streaming_callback:
      tools:
      response_format:

connections:  # Defines how the components are connected
- sender: bm25_retriever.documents
  receiver: document_joiner.documents
- sender: query_embedder.embedding
  receiver: embedding_retriever.query_embedding
- sender: embedding_retriever.documents
  receiver: document_joiner.documents
- sender: document_joiner.documents
  receiver: ranker.documents
- sender: ranker.documents
  receiver: meta_field_grouping_ranker.documents
- sender: meta_field_grouping_ranker.documents
  receiver: answer_builder.documents
- sender: meta_field_grouping_ranker.documents
  receiver: ChatPromptBuilder.documents
- sender: OutputAdapter.output
  receiver: answer_builder.replies
- sender: OllamaChatGenerator.replies
  receiver: OutputAdapter.replies
- sender: ChatPromptBuilder.prompt
  receiver: OllamaChatGenerator.messages

inputs:  # Define the inputs for your pipeline
  query:  # These components will receive the query as input
  - "bm25_retriever.query"
  - "query_embedder.text"
  - "ranker.query"
  - "answer_builder.query"
  - "ChatPromptBuilder.query"
  filters:  # These components will receive a potential query filter as input
  - "bm25_retriever.filters"
  - "embedding_retriever.filters"

outputs:  # Defines the output of your pipeline
  documents: "meta_field_grouping_ranker.documents"  # The output of the pipeline is the retrieved documents
  answers: "answer_builder.answers"  # The output of the pipeline is the generated answers

max_runs_per_component: 100

metadata: {}

Parameters​

Inputs​

ParameterTypeDefaultDescription
messagesList[ChatMessage]A list of ChatMessage instances representing the input messages.
generation_kwargsOptional[Dict[str, Any]]NonePer-call overrides for Ollama inference options. These are merged on top of the instance-level generation_kwargs. For a full list, see the Ollama documentation.
toolsOptional[Union[List[Tool], Toolset]]NoneA list of tools or a Toolset for which the model can prepare calls.
streaming_callbackOptional[Callable[[StreamingChunk], None]]NoneA callable to receive StreamingChunk objects as they arrive. Supplying a callback switches the component into streaming mode.

Outputs​

ParameterTypeDefaultDescription
repliesList[ChatMessage]A list of ChatMessages containing the model's response.

Init Parameters​

These are the parameters you can configure in Pipeline Builder:

ParameterTypeDefaultDescription
modelstrorca-miniThe name of the model to use. The model must already be present (pulled) in the running Ollama instance.
urlstrhttp://localhost:11434The base URL of the Ollama server.
generation_kwargsOptional[Dict[str, Any]]NoneOptional arguments to pass to the Ollama generation endpoint, such as temperature, top_p, and others. See the available arguments in Ollama documentation.
timeoutint120The number of seconds before throwing a timeout error from the Ollama API.
keep_aliveOptional[Union[float, str]]NoneControls how long the model will stay loaded into memory following the request. If not set, it will use the default value from Ollama (five minutes).
streaming_callbackOptional[Callable[[StreamingChunk], None]]NoneA callback function that is called when a new token is received from the stream.
toolsOptional[Union[List[Tool], Toolset]]NoneA list of haystack.tools.Tool or a haystack.tools.Toolset. Not all models support tools. For a list of models compatible with tools, see the models page.
response_formatOptional[Union[None, Literal['json'], JsonSchemaValue]]NoneThe format for structured model outputs. The value can be: None (no specific format), "json" (JSON object), or a JSON Schema.
thinkboolFalseWhether to enable thinking mode for models that support it.

Run Method Parameters​

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

ParameterTypeDefaultDescription
messagesList[ChatMessage]A list of ChatMessage instances representing the input messages.
generation_kwargsOptional[Dict[str, Any]]NonePer-call overrides for Ollama inference options. These are merged on top of the instance-level generation_kwargs. For a complete list of arguments, see the Ollama documentation.
toolsOptional[Union[List[Tool], Toolset]]NoneA list of tools or a Toolset for which the model can prepare calls.
streaming_callbackOptional[Callable[[StreamingChunk], None]]NoneA callable to receive StreamingChunk objects as they arrive. Supplying a callback switches the component into streaming mode.