AIMLAPIChatGenerator

Generate chat responses using models hosted on the AI/ML API platform. AI/ML API provides access to hundreds of AI models through a single OpenAI-compatible API.

Key Features

• Access to 200+ AI models from OpenAI, Anthropic, Meta, Mistral, and others through a single endpoint.
• Compatible with OpenAI's Chat Completions API format.
• Supports streaming responses through a configurable callback.
• Supports tool calling for agentic workflows.
• Configurable retry logic and timeout settings.

Configuration

Add Workspace-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Workspace>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Organization>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.

1. Drag the AIMLAPIChatGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to the model you want to use (for example, openai/gpt-4o or anthropic/claude-opus-4-5). For a full list, see the AI/ML API model catalog.
   2. Create a secret with your AI/ML API key and set it as api_key. Use AIMLAPI_API_KEY as the environment variable name. For instructions, see Create Secrets.
4. Go to the Advanced tab to configure generation_kwargs such as max_tokens, temperature, and top_p.

Connections

AIMLAPIChatGenerator receives a list of ChatMessage objects as input, typically from PromptBuilder or ChatPromptBuilder. It outputs a list of reply ChatMessage objects you can connect to AnswerBuilder or other downstream components.

Source Code

To check this component's source code, open chat_generator.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  AIMLAPIChatGenerator:
    type: haystack_integrations.components.generators.aimlapi.chat.chat_generator.AIMLAPIChatGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - AIMLAPI_API_KEY
        strict: false
      model: openai/gpt-4o
      generation_kwargs:
        max_tokens: 1024
        temperature: 0.7

Using the Component in a Pipeline

# haystack-pipeline
components:
  prompt_builder:
    type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
    init_parameters:
      required_variables: "*"
      template:
      - role: system
        content: You are a helpful assistant. Answer questions based on the provided documents.
      - role: user
        content: |
          Documents:
          {% for doc in documents %}
          {{ doc.content }}
          {% endfor %}
          Question: {{ question }}

  llm:
    type: haystack_integrations.components.generators.aimlapi.chat.chat_generator.AIMLAPIChatGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - AIMLAPI_API_KEY
        strict: false
      model: openai/gpt-4o
      generation_kwargs:
        max_tokens: 1024

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: prompt_builder.prompt
  receiver: llm.messages
- sender: llm.replies
  receiver: answer_builder.replies

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - answer_builder.query
  - prompt_builder.question
  documents:
  - prompt_builder.documents

outputs:
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
messages	List[ChatMessage]	A list of chat messages representing the conversation so far.

Outputs

Parameter	Type	Description
replies	List[ChatMessage]	A list of generated reply messages from the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var("AIMLAPI_API_KEY")	The AI/ML API key.
model	str	openai/gpt-5-chat-latest	The model to use. Supports any model available in the AI/ML API catalog.
api_base_url	Optional[str]	https://api.aimlapi.com/v1	The base URL for the AI/ML API.
streaming_callback	Optional[Callable]	None	A callback function for streaming responses. When set, the model streams tokens as they're generated.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters passed to the API, such as max_tokens, temperature, and top_p.
tools	Optional[List[Tool]]	None	A list of tools the model can use.
timeout	Optional[float]	None	Request timeout in seconds.
max_retries	Optional[int]	None	Maximum number of retries on API errors. Falls back to AIMLAPI_MAX_RETRIES env var or 5.
extra_headers	Optional[Dict[str, Any]]	None	Additional HTTP headers to include in requests.
http_client_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments to pass to the HTTP client.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
messages	List[ChatMessage]		A list of chat messages representing the conversation.
streaming_callback	Optional[Callable]	None	A callback function to override the init-time streaming callback.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional generation parameters to override init-time values.
tools	Optional[List[Tool]]	None	Tools to make available to the model, overriding init-time tools.

Related Information

• Pipeline Components