FallbackChatGenerator

Use FallbackChatGenerator to try multiple chat generators sequentially. If one generator fails, it falls back to the next one in the list. If all generators fail, it raises an error with all the details.

Key Features

• Tries multiple chat generators in order, automatically falling back when one fails.
• Supports any exception type as a trigger for fallover, including timeouts, rate limits, and server errors.
• Forwards all parameters transparently to each underlying generator.
• Returns metadata about which generator succeeded and which ones failed.
• Compatible with any component that accepts ChatMessage objects.

Configuration

1. Drag the FallbackChatGenerator component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. On the General tab:
   1. Add the list of chat generators to try in order. Each entry requires a type and init_parameters.
4. Go to the Advanced tab to configure timeout settings on individual generators.

Connections

FallbackChatGenerator receives a messages list of ChatMessage objects — typically from ChatPromptBuilder. It outputs replies (a list of ChatMessage objects from the first successful generator) and meta (execution metadata including which generator succeeded). Connect its replies output to AnswerBuilder or OutputAdapter for downstream processing.

Usage Example

This query pipeline uses FallbackChatGenerator to try GPT-4o first and fall back to GPT-4o-mini if it fails:

components:
  ChatPromptBuilder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      template:
        - _content:
            - text: "You are a helpful assistant that answers questions concisely."
          _role: system
        - _content:
            - text: "Question: {{ question }}"
          _role: user
      required_variables:
      variables:

  FallbackChatGenerator:
    type: haystack.components.generators.chat.fallback.FallbackChatGenerator
    init_parameters:
      chat_generators:
        - type: haystack.components.generators.chat.openai.OpenAIChatGenerator
          init_parameters:
            model: gpt-4o
            generation_kwargs:
              temperature: 0.7
              max_tokens: 500
        - type: haystack.components.generators.chat.openai.OpenAIChatGenerator
          init_parameters:
            model: gpt-4o-mini
            generation_kwargs:
              temperature: 0.7
              max_tokens: 500

  AnswerBuilder:
    type: haystack.components.builders.answer_builder.AnswerBuilder
    init_parameters:
      pattern:
      reference_pattern:

connections:
  - sender: ChatPromptBuilder.prompt
    receiver: FallbackChatGenerator.messages
  - sender: FallbackChatGenerator.replies
    receiver: AnswerBuilder.replies

inputs:
  query:
    - ChatPromptBuilder.question
    - AnswerBuilder.query

outputs:
  answers: AnswerBuilder.answers

Parameters

Inputs

Parameter	Type	Default	Description
messages	List[ChatMessage]		The conversation history as a list of ChatMessage objects.
generation_kwargs	Optional[Dict[str, Any]]	None	Optional parameters for the chat generator (for example, temperature, max_tokens).
tools	Optional[ToolsType]	None	A list of Tool or Toolset objects the generator can use.
streaming_callback	Optional[StreamingCallbackT]	None	Optional callable for handling streaming responses.

Outputs

Parameter	Type	Default	Description
replies	List[ChatMessage]		Generated ChatMessage objects from the first successful generator.
meta	Dict[str, Any]		Execution metadata including successful_chat_generator_index, successful_chat_generator_class, total_attempts, failed_chat_generators, and any metadata from the successful generator.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
chat_generators	List[ChatGenerator]		A list of chat generator components to try in order.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		The conversation history as a list of ChatMessage objects.
generation_kwargs	Optional[Dict[str, Any]]	None	Optional parameters for the chat generator.
tools	Optional[ToolsType]	None	A list of Tool and/or Toolset objects for the generators to use.
streaming_callback	Optional[StreamingCallbackT]	None	Optional callable for handling streaming responses.