LiteLLMChatGenerator

Generate chat responses using any of 100+ LLM providers through LiteLLM's unified interface.

Key Features

• Single component for access to OpenAI, Anthropic, Google, AWS Bedrock, Azure, Cohere, Mistral, Groq, and many more providers.
• Uses LiteLLM's unified API format — no need to switch components when changing providers.
• Model names use the LiteLLM format provider/model-name (for example, anthropic/claude-sonnet-4-20250514).
• Accepts and returns messages in ChatMessage format.
• Supports streaming responses through a configurable callback.
• Supports tool calling for agentic workflows.

Configuration

1. Drag the LiteLLMChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model using LiteLLM format: provider/model-name (for example, openai/gpt-4o or anthropic/claude-opus-4-5). For the full list of supported providers and model names, see the LiteLLM documentation.
   2. Create a secret with the appropriate API key for your provider (for example, OPENAI_API_KEY, ANTHROPIC_API_KEY). For instructions, see Create Secrets.
4. Go to the Advanced tab to configure generation_kwargs, api_base_url, and tools.

Connections

LiteLLMChatGenerator receives a list of ChatMessage objects, typically from PromptBuilder or ChatPromptBuilder. It outputs a list of reply ChatMessage objects you can connect to AnswerBuilder or other downstream components.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  LiteLLMChatGenerator:
    type: haystack_integrations.components.generators.litellm.chat.chat_generator.LiteLLMChatGenerator
    init_parameters:
      model: openai/gpt-4o
      generation_kwargs:
        temperature: 0.7
        max_tokens: 1024

Using the Component with Different Providers

  # Using Anthropic
  LiteLLMChatGenerator:
    type: haystack_integrations.components.generators.litellm.chat.chat_generator.LiteLLMChatGenerator
    init_parameters:
      model: anthropic/claude-opus-4-5
      generation_kwargs:
        max_tokens: 2048

  # Using AWS Bedrock
  LiteLLMChatGenerator:
    type: haystack_integrations.components.generators.litellm.chat.chat_generator.LiteLLMChatGenerator
    init_parameters:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      generation_kwargs:
        max_tokens: 1024

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: user
        content: "Answer the following question: {{ question }}"

  llm:
    type: haystack_integrations.components.generators.litellm.chat.chat_generator.LiteLLMChatGenerator
    init_parameters:
      model: openai/gpt-4o-mini
      generation_kwargs:
        max_tokens: 1024

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Optional[Secret]	None	The API key for the provider. Set the appropriate environment variable (for example, OPENAI_API_KEY) and use Secret.from_env_var().
model	str	openai/gpt-4o	The model to use in LiteLLM format: provider/model-name. See the LiteLLM provider list.
streaming_callback	Optional[Callable]	None	A callback function for streaming responses.
api_base_url	Optional[str]	None	A custom API base URL for proxies or custom deployments.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters, such as temperature, max_tokens, and top_p.
tools	Optional[List[Tool]]	None	A list of tools the model can use for tool calling.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
streaming_callback	Optional[Callable]	None	A callback function to override the init-time streaming callback.
generation_kwargs	Optional[Dict[str, Any]]	None	Generation parameters to override init-time values.
tools	Optional[List[Tool]]	None	Tools to make available to the model.

Related Information

• Pipeline Components