FallbackChatGenerator

Use FallbackChatGenerator to try multiple chat generators sequentially. If one generator fails, it falls back to the next one in the list. If all generators fail, it raises an error with all the details.

Key Features

• Tries multiple chat generators in order and returns the first successful result.
• Automatically triggers failover on any exception, including timeout, rate limit, authentication, context length, and server errors.
• Forwards all parameters transparently to the underlying chat generators.
• Returns detailed error information if all generators fail.
• Lets you control per-generator timeout through each generator's timeout init parameter.

Configuration

1. Drag the FallbackChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   • Add a list of chat generators to try in order under chat_generators. The component tries each one in sequence until one succeeds.

To control the timeout of each generator, set the timeout parameter in that generator's init parameters in the YAML.

Connections

FallbackChatGenerator accepts a list of ChatMessage objects as input. Connect it to any component that produces messages, such as ChatPromptBuilder.

It outputs replies as a list of ChatMessage objects from the first successful generator. Connect its replies output to any component that consumes replies, such as DeepsetAnswerBuilder or OutputAdapter.

Source Code

To check this component's source code, open fallback.py in the Haystack repository.

Usage Examples

Basic Configuration

  FallbackChatGenerator:
    type: haystack.components.generators.chat.fallback.FallbackChatGenerator
    init_parameters:
      chat_generators:
      - type: haystack.components.generators.chat.openai.OpenAIChatGenerator
        init_parameters:
          model: gpt-4o
          generation_kwargs:
            temperature: 0.7
            max_tokens: 500
      - type: haystack.components.generators.chat.openai.OpenAIChatGenerator
        init_parameters:
          model: gpt-4o-mini
          generation_kwargs:
            temperature: 0.7
            max_tokens: 500

This query pipeline uses FallbackChatGenerator to try GPT-4o first and fall back to GPT-4o-mini if it fails:

components:
  ChatPromptBuilder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      template:
        - _content:
            - text: "You are a helpful assistant that answers questions concisely."
          _role: system
        - _content:
            - text: "Question: {{ question }}"
          _role: user
      required_variables:
      variables:

  FallbackChatGenerator:
    type: haystack.components.generators.chat.fallback.FallbackChatGenerator
    init_parameters:
      chat_generators:
        - type: haystack.components.generators.chat.openai.OpenAIChatGenerator
          init_parameters:
            model: gpt-4o
            generation_kwargs:
              temperature: 0.7
              max_tokens: 500
        - type: haystack.components.generators.chat.openai.OpenAIChatGenerator
          init_parameters:
            model: gpt-4o-mini
            generation_kwargs:
              temperature: 0.7
              max_tokens: 500

  AnswerBuilder:
    type: haystack.components.builders.answer_builder.AnswerBuilder
    init_parameters:
      pattern:
      reference_pattern:

connections:
  - sender: ChatPromptBuilder.prompt
    receiver: FallbackChatGenerator.messages
  - sender: FallbackChatGenerator.replies
    receiver: AnswerBuilder.replies

inputs:
  query:
    - ChatPromptBuilder.question
    - AnswerBuilder.query

outputs:
  answers: AnswerBuilder.answers

Parameters

Inputs

Parameter	Type	Default	Description
messages	List[ChatMessage]		The conversation history as a list of ChatMessage objects.
generation_kwargs	Optional[Dict[str, Any]]	None	Optional parameters for the chat generator (for example, temperature, max_tokens).
tools	Optional[ToolsType]	None	A list of Tool or Toolset objects the generator can use.
streaming_callback	Optional[StreamingCallbackT]	None	Optional callable for handling streaming responses.

Outputs

Parameter	Type	Default	Description
replies	List[ChatMessage]		Generated ChatMessage objects from the first successful generator.
meta	Dict[str, Any]		Execution metadata including successful_chat_generator_index, successful_chat_generator_class, total_attempts, failed_chat_generators, and any metadata from the successful generator.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
chat_generators	List[ChatGenerator]		A list of chat generator components to try in order.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		The conversation history as a list of ChatMessage objects.
generation_kwargs	Optional[Dict[str, Any]]	None	Optional parameters for the chat generator.
tools	Optional[ToolsType]	None	A list of Tool and/or Toolset objects for the generators to use.
streaming_callback	Optional[StreamingCallbackT]	None	Optional callable for handling streaming responses.