Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

LLMs in Your Pipelines

To generate text, answer questions, or summarize documents in a pipeline, you use a large language model (LLM). You add the LLM component to your pipeline, choose a model, and tell it what to do in a prompt. This page explains how that works and what a ChatMessage is, the format the model uses to receive and return messages.

Use a Model with the LLM Component​

The LLM component is the way to use a model in a pipeline. It works with any model provider, such as OpenAI, Azure, or Amazon Bedrock. You send it messages, it passes them to the model you chose, and it returns the model's answer.

To use it, you do three things:

  1. Connect Haystack Platform to your model provider. For details, see Add Integrations.
  2. Add the LLM component to your pipeline and choose the model from the list of models.
  3. Write a system prompt and a user prompt. The user prompt can contain variables, such as the user's question or retrieved documents, that the pipeline fills in at query time.

For configuration steps and parameters, see LLM. For help with choosing a model, see Language Models in Haystack Enterprise Platform.

LLM or Agent​

Use LLM when you want the model to generate text from a prompt, for example to answer a question from retrieved documents. If the model needs to call tools or work through several steps, use the Agent component instead.

Write Your Prompts​

The system prompt sets the model's role and rules, for example "You are a helpful assistant answering questions about the documents." The user prompt carries the input for the current query. You write both in Jinja2 syntax and wrap each one in a message tag:

{% message role="system" %}
You are a helpful assistant answering the user's questions.
If the answer is not in the documents, say so.
{% endmessage %}

For tips on writing good prompts, see Writing Prompts in Haystack Enterprise Platform.

Use the Answer in Your Pipeline​

LLM returns its answer as a ChatMessage through the last_message output. If the model's answer is the final result of your pipeline, connect last_message to the Output component. You don't need an extra component to convert it. Connections between string and ChatMessage convert automatically in both directions. For details, see Smart Connections.

For typical setups, such as retrieval followed by an LLM, see Common Component Combinations.

Streaming​

Streaming refers to the process of generating responses in real time as the model processes input. Instead of waiting for the entire input to be processed before responding, the model generates an answer token by token, making the communication feel more fluid and immediate.

Streaming is particularly useful in applications where timely feedback is crucial, such as in live chat interfaces or conversational agents.

All RAG pipelines in Haystack Enterprise Platform have streaming enabled by default.

To learn more, see Enable Streaming.

Understand ChatMessage​

A ChatMessage is how you and the model exchange messages. The LLM component takes messages as input and returns messages as output, so it helps to know what one contains.

Each message has a role and content:

  • Role: who the message is from. The system role sets the tone and instructions for the conversation. The user role holds the input, usually a query, and can also hold documents you want the model to use. The assistant role is the model's answer. The tool role holds the result of a tool call.
  • Content: what the message says. In most cases, the content is text. A message can also hold other content types, such as images, a tool call, or a tool call result.

A conversation is a list of messages. The model reads the whole list and adds its answer as the next assistant message. This is how it keeps the context of a conversation.

If the model calls a tool, its message has the content type tool_call. You can use this to create different paths in your pipeline. For example, use ConditionalRouter to send the pipeline one way when the model makes a tool call and another way when it doesn't.

For details on the message format and its properties, see ChatMessage in Haystack documentation.

If Your Pipeline Uses ChatGenerators or Generators​

Older pipelines use a ChatGenerator with a ChatPromptBuilder, or a legacy Generator with a PromptBuilder, to talk to a model. LLM replaces all of them, and Haystack 3 removes the legacy Generators. If you have a pipeline like this, consider moving to LLM. For details, see Simplify Your Pipelines with Smart Connections.

Until you move, keep these differences in mind:

  • Generators take a string prompt from PromptBuilder and return a list of strings. They don't support tool calling.
  • ChatGenerators take a list of ChatMessage objects, or a Jinja2 string, from ChatPromptBuilder and return a list of ChatMessage objects. They support multi-turn conversations and tool calling.
  • Each model provider has its own ChatGenerator. For example, to use Claude through Anthropic's API, choose AnthropicChatGenerator. To use models through Amazon Bedrock, choose AmazonBedrockChatGenerator.
  • To pass the output of a ChatGenerator to AnswerBuilder, you can connect them directly. Smart connections convert the ChatMessage to a string for you.

When you write instructions in the template parameter of ChatPromptBuilder, use this format:

- content: 
    - content_type:  # replace this with the content type, supported content types are: text, tool_call, tool_call_result
      # content may contain variables
  role: role # supported roles are: user, system, assistant, tool`

For example, this template contains a system message and a user message with the query:

- _content:
    - text: |
        You are a helpful assistant answering the user's questions.
        If the answer is not in the documents, rely on the web_search tool to find information.
        Do not use your own knowledge.
  _role: system
- _content:
    - text: |
        Question: {{ query }}
  _role: user

You can also pass the template as a Jinja2 string with the message tag, as in the LLM examples above. For details, see ChatPromptBuilder.

Testing Prompts

You can test a ChatGenerator's prompt in Prompt Explorer only if you use Jinja2 syntax in the template parameter. If you write the template as ChatMessage objects, experiment with prompts in the Playground instead. Use the Configurations feature to change the ChatPromptBuilder's template parameter. For details, see Modify Pipeline Parameters at Query Time.