Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Add Custom Model Definitions

Configure custom model configurations to use in your pipelines. Custom models let you define YAML configurations for models and reuse them across pipelines.

About This Task​

With custom models, you can:

  • Use models from providers beyond the standard integrations.
  • Configure model parameters and generation settings for your use case.
  • Create reusable model configurations for your organization or workspace.
  • Set custom API endpoints and authentication methods.

A custom model definition is a reusable configuration for a model. It can include a specific model, its parameters, or both. For example, you can define an OpenAI model with custom parameters and reuse it across multiple pipelines.

After you save a custom model, it appears in the list of available models on components that support models, such as LLM or Agent.

You can create custom models at two levels:

  • Workspace: Available only in the current workspace.
  • Organization: Available across all workspaces in your organization.

Prerequisites​

  • You must be an organization Admin to add custom model definitions for an organization, or have write access to a workspace to add custom model definitions to a workspace.
  • You must have a valid API key from the model provider.

Add a Custom Model Definition​

  1. In Haystack Enterprise Platform, click your profile icon and choose Settings.
  2. Go to Organization>Integrations (for organization-level custom models) or Workspace>Integrations (for workspace-level custom models).
  3. Find Custom Models and click Configure.
  4. Click Add model.
  5. Enter the following details:
    • Model name: A descriptive name for your custom model
    • Provider: Pick the model provider from the list. For a provider that is not listed, choose Custom.
  6. Configure the YAML settings:
    • Chat Generator: Define the component configuration including API endpoints and authentication.
    • Model Parameters: Specify model parameters, like temperature and max output tokens. These parameters are specific to the model provider you choose. Check the model provider's documentation for allowed parameters.
  7. Optionally, try the model to check if the configuration is working.
  8. Save your changes.

Use Custom Models in Pipelines​

Custom Models Are Copied Into Pipelines

When you add a custom model to your pipeline, it creates a copy of that model. This means any changes you make to the model definition on the Integrations page do not affect the model in your pipeline. To update the model in your pipeline, edit the pipeline YAML configuration.

After creating a custom model definition, you can use it in your pipelines by choosing it from a list of available models when configuring AI components, like LLM or Agent. The model is shown alongside standard models in component configuration.

Custom model on the model list

Manage Custom Models​

You can update or delete custom model definitions. Changes you make to the model definition do not affect the model in your pipelines. To update the model in your pipeline, edit the pipeline YAML configuration.

  1. On the Integrations page, click Configure on the Custom Models Card. All custom model definitions are listed here.
  2. Click Edit next to the model definition you want to update, or Delete to remove it.

Duplicate Models and Overrides​

If saving fails because the model duplicates another one, or you want a workspace-specific version of an organization model, these rules apply. A workspace model can have an organization-level counterpart: a model with the same provider component and model name. As with secrets and integrations, the workspace model overrides its organization counterpart, so you can keep editing the workspace model.

The platform checks for duplicates when you save a model:

  • Adding an organization model: The platform compares it only with other organization models. A workspace model with the same component and model name doesn't block it.
  • Adding a workspace model: Saving fails if the same component and model name already exist in that workspace or at the organization level. To resolve it, change the model name or edit the existing model instead.
  • Editing a model: The platform checks for duplicates only if you change the provider component or the model name. Changes to other settings, such as the display name or model parameters, are always allowed, even if the model has a counterpart at another level. If you change the component or model name so that a workspace model matches another model in its workspace or at the organization level, saving fails. An organization model is compared only with other organization models.

Find Models That Hide Other Models​

If a custom model doesn't behave as expected, check whether it overrides another model with the same provider component and model name. To find out, go to Settings > Organization>Integrations or Workspace>Integrations, click Configure on Custom Models, and check the Name column. A tag under the model name shows what the model hides:

  • Hides organization model: Shown in the workspace list. An organization custom model has the same provider component and model name, so this workspace model takes precedence over it in the workspace.
  • Hides platform model: Shown in the workspace and organization lists. A platform model has the same provider component and model name, so this custom model takes precedence over it.

A model can show both tags in the workspace list.

Two models match only if they use the same provider component and the same model name or, when a model has no model name, the same Azure deployment name. The display name doesn't matter, and a model without a model name never matches.

The organization list doesn't show which workspace models override an organization model.

Retry When Another Change Is in Progress​

The platform saves model changes in an organization one at a time, so two people can't create the same model at once. If another model change in your organization is still running, your save can fail with HTTP status 503 and a message that another change is in progress. Wait about 5 seconds and save again. If you call the API directly, the response includes a Retry-After header with the number of seconds to wait. For the endpoints, see Create Organization Model, Update Organization Model, and Create Workspace Model.

If the error keeps appearing after several retries, a model change in your organization may be stuck. Contact support.