OrcaRouterChatGenerator

Generate chat responses using large language models routed through OrcaRouter. This component extends OpenAIChatGenerator and exposes 100+ chat models from providers such as OpenAI, Anthropic, Google, DeepSeek, and Qwen behind a single OpenAI-compatible endpoint and API key.

Key Features

• Routes requests to upstream models using a provider/model namespace (for example, openai/gpt-4o-mini or anthropic/claude-opus-4.8).
• Supports the orcarouter/auto router to let OrcaRouter select a model per request.
• Supports streaming, tool calling, and structured outputs through the OpenAI-compatible API.
• Accepts all parameters supported by the OrcaRouter chat completion endpoint via generation_kwargs.

Configuration

Add Workspace-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Workspace>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Organization>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.

1. Drag the OrcaRouterChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Connect Haystack Platform to your OrcaRouter account by creating a secret called ORCAROUTER_API_KEY. For more information about secrets, see Create Secrets.
   2. Select the model to use. The default is openai/gpt-4o-mini.
4. Go to the Advanced tab to configure generation_kwargs, timeout, max_retries, http_client_kwargs, and tool options.

Connections

OrcaRouterChatGenerator receives rendered chat prompts from ChatPromptBuilder through its messages input. It sends generated replies through its replies output to downstream components such as AnswerBuilder or OutputAdapter.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  OrcaRouterChatGenerator:
    type: haystack_integrations.components.generators.orcarouter.chat.chat_generator.OrcaRouterChatGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - ORCAROUTER_API_KEY
        strict: false
      model: openai/gpt-4o-mini
      api_base_url: https://api.orcarouter.ai/v1

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: system
        content: You are a helpful assistant.
      - role: user
        content: "{{ question }}"

  llm:
    type: haystack_integrations.components.generators.orcarouter.chat.chat_generator.OrcaRouterChatGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - ORCAROUTER_API_KEY
        strict: false
      model: openai/gpt-4o-mini
      generation_kwargs:
        max_tokens: 512
        temperature: 0.7

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var("ORCAROUTER_API_KEY")	The OrcaRouter API key.
model	str	openai/gpt-4o-mini	The OrcaRouter chat completion model. Models use a provider/model namespace. Use orcarouter/auto to let OrcaRouter route the request.
streaming_callback	Optional[StreamingCallbackT]	None	A callback function called when a new token is received from the stream.
api_base_url	Optional[str]	https://api.orcarouter.ai/v1	The OrcaRouter API base URL.
organization	Optional[str]	None	Your OrcaRouter organization ID, if any.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional parameters sent to the OrcaRouter endpoint, such as max_tokens, temperature, top_p, stream, and extra_body.
tools	Optional[ToolsType]	None	A list of tools or a Toolset the model can use.
tools_strict	bool	False	Whether to enable strict schema adherence for tool calls.
timeout	Optional[float]	None	The timeout for the OrcaRouter API call.
max_retries	Optional[int]	None	Maximum number of retries after an internal error. Defaults to the OPENAI_MAX_RETRIES environment variable or five.
http_client_kwargs	Optional[Dict[str, Any]]	None	Keyword arguments to configure a custom httpx client.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters to override init-time values.
streaming_callback	Optional[StreamingCallbackT]	None	A callback function to override the init-time streaming callback.
tools	Optional[ToolsType]	None	Tools to make available to the model, overriding init-time tools.

Related Information

• OpenRouterChatGenerator
• Pipeline Components