Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

GatewayBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Gateway is where you publish your pipelines to production, giving your callers a stable endpoint that doesn't change when you update the pipeline behind it.

What Is Gateway?​

Gateway separates work in progress from what is live in production. You build and iterate in the Builder, and when a pipeline is ready, you publish it through Gateway. Callers connect to what you publish, not to the pipeline itself. This way, the connection stays stable no matter how many times you update or rebuild the pipeline behind it.

Gateway lives under its own section in the navigation, where you can manage everything you've published from your current workspace.

Permissions​

Accessing Gateway requires the pipelines: read permission, the same permission used for the Pipelines page.

How Gateway Fits into Your Workflow​

Haystack Enterprise Platform follows a build → test → publish journey:

  1. Build: Create and iterate on your pipeline in the Builder.
  2. Test: Try your pipeline in the Playground or Prompt Explorer.
  3. Publish: Use Gateway to pin a pipeline version and expose a stable endpoint.

Services​

Inside Gateway, you publish a pipeline by creating a service. A service is a stable, production-ready endpoint that exposes one specific pipeline version to your callers.

A service is a stable environment with set replica and idle timeout configurations. Publishing a pipeline to a service ensures that the pipeline's availability and performance stay consistent. You can keep updating the pipeline without interrupting the service.

You can publish one pipeline version to a service at a time. To publish multiple pipelines through Gateway, create one service for each pipeline.

You can also publish the same pipeline to multiple services. This is useful if you want different service levels for different use cases, or if you want to publish different pipeline versions at once.

You publish a pipeline to a service from the Builder, where you choose the version to publish and either select an existing service or create a new one.

Main Benefits​

  • Stable production endpoints: Callers connect to a service, not a pipeline. The pipeline can keep changing underneath without affecting what is live.
  • Safe iteration: Experiment freely on your pipeline without touching what your users depend on.
  • Version control with intent: Promote a pipeline version to a service when it is ready, instead of relying on auto-generated history.
  • One pipeline, multiple services: Dev, staging, and production each independently managed from the same pipeline.

Publishing to a Service vs. Deploying a Pipeline​

You can make a pipeline available in two ways: by publishing it to a service, or by deploying it without a service:

  • Deploying a pipeline makes it available for queries directly. It's best for quick testing and internal use, where endpoint stability is not a priority. When you update the pipeline and want to deploy a new version, the endpoint URL will change. Deploying a pipeline is required to test it in the Playground.

  • (Recommended) Publishing to a service assigns a specific pipeline version to a stable endpoint. The endpoint doesn't change when you update the pipeline. This is the recommended approach for production integrations, where your callers need a consistent endpoint they can rely on.

When building a pipeline other systems will connect to, use a service.

Service Metrics​

Clicking a service in Gateway opens the Service Overview page where you can review service performance and history. You can switch between the sections to check how to connect to the service, see its traces, and update the service settings.

Service Traces​

You can view the traces for a service on the Service Traces page. You can filter the traces status, source, latency, and more. For details, see Traces and Trace Your Pipeline.

Hosting Settings​

When you open a service and switch to the Settings tab, you land on a page where you can review and update all service settings, including hosting.

You can configure the hosting settings for a service to define how scalable and reliable it is. This includes:

  • Minimum and maximum number of replicas available. Replicas are duplicate versions of a pipeline that are always available to handle traffic. When the traffic volume increases, Haystack Platform seamlessly switches to a functioning replica to maintain uninterrupted service.
  • Idle timeout. The time after which a service enters standby mode to save resources. A pipeline gets automatically activated if you start searching with it, or you can activate it manually on the Pipelines page. When a pipeline is idle, your data remains perfectly safe and accessible anytime.

For production and business-critical use cases, we recommend the default hosting settings: 2 replicas at minimum and 24 hours idle timeout. For testing and low demand pipelines, we recommend 0 minimum replicas and 20 minutes idle timeout to save resources.

Service and Pipeline Hosting Settings​

Hosting settings for a service are separate from the pipeline's hosting settings. When you publish a pipeline to a service, it uses the service's hosting settings. When you deploy a pipeline without a service, it uses the pipeline's hosting settings configured in pipeline settings. For details, see Pipeline Hosting Settings.

Service Status​

Possible statuses of a service are:

  • Active — the service is deployed and ready to handle requests.
  • Standby — the service is idle to save resources. It reactivates automatically when a request arrives, or you can reactivate it manually. For details, see Manage a Service.
  • Deploying — the service is being deployed.
  • Failed — the service could not be deployed or encountered a fatal error.
  • Inactive — the service has been deactivated.

You can deactivate a service from the Service Details page. To activate it again, use Activate service on that page.