LlamaCppChatGenerator

Generate chat responses using LLMs run locally via llama.cpp.

Key Features

• Runs GGUF-format language models locally without requiring cloud API access.
• Works on standard hardware, including machines without dedicated GPUs.
• Accepts and returns messages in ChatMessage format.
• Supports streaming responses through a configurable callback.
• Supports tool calling for agentic workflows.
• Supports multimodal models (text and image) through the LLaVA family and similar models.
• Configurable context length and batch size.

Configuration

1. Drag the LlamaCppChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to the path or URL of the GGUF model file to load (for example, /path/to/mistral-7b.gguf).
4. Go to the Advanced tab to configure n_ctx, n_batch, model_kwargs, generation_kwargs, and streaming_callback.

Connections

LlamaCppChatGenerator receives a list of ChatMessage objects, typically from PromptBuilder or ChatPromptBuilder. It outputs a list of reply ChatMessage objects you can connect to AnswerBuilder or other downstream components.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  LlamaCppChatGenerator:
    type: haystack_integrations.components.generators.llama_cpp.LlamaCppChatGenerator
    init_parameters:
      model: /path/to/mistral-7b-instruct.gguf
      n_ctx: 4096
      generation_kwargs:
        max_tokens: 1024
        temperature: 0.7

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: user
        content: "Answer the following question: {{ question }}"

  llm:
    type: haystack_integrations.components.generators.llama_cpp.LlamaCppChatGenerator
    init_parameters:
      model: /path/to/mistral-7b-instruct.gguf
      n_ctx: 4096
      generation_kwargs:
        max_tokens: 1024

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
model	str	(required)	The path to the GGUF model file or a URL to download the model from.
n_ctx	Optional[int]	0	The maximum context window size. Set to 0 to use the model's default.
n_batch	Optional[int]	512	The number of tokens to process in parallel during evaluation.
model_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for the llama_cpp.Llama constructor.
generation_kwargs	Optional[Dict[str, Any]]	None	Default generation parameters such as max_tokens, temperature, top_p, top_k, and stop.
tools	Optional[List[Tool]]	None	A list of tools the model can use for tool calling.
streaming_callback	Optional[Callable]	None	A callback function for streaming token-by-token responses.
chat_handler_name	Optional[str]	None	The name of a registered chat handler for multimodal models.
model_clip_path	Optional[str]	None	The path to the CLIP model for multimodal (image + text) support.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
generation_kwargs	Optional[Dict[str, Any]]	None	Generation parameters to override init-time values.
tools	Optional[List[Tool]]	None	Tools to make available to the model.

Related Information

• Pipeline Components