GatewayBeta
Gateway is where you publish your pipelines to production, giving your callers a stable endpoint that doesn't change when you update the pipeline behind it.
What Is Gateway?
Gateway separates work in progress from what is live in production. You build and iterate in the Builder, and when a pipeline is ready, you publish it through Gateway. Callers connect to what you publish, not to the pipeline itself. This way, the connection stays stable no matter how many times you update or rebuild the pipeline behind it.
Gateway lives under its own section in the navigation, where you can manage everything you've published from your current workspace.
Permissions
To access Gateway, you need the services: read permission. To create, update, or delete services, publish pipeline versions to services, or configure service hosting, you need the services: write permission.
services: write permission alone isn't enough.How Gateway Fits into Your Workflow
Haystack Enterprise Platform follows a build → test → publish journey:
- Build: Create and iterate on your pipeline in the Builder.
- Test: Try your pipeline in the Playground or Prompt Explorer.
- Publish: Use Gateway to pin a pipeline version and expose a stable endpoint.
Services
Inside Gateway, you publish a pipeline by creating a service. A service is a stable, production-ready endpoint that exposes one specific pipeline version to your callers. Publishing a pipeline to a service ensures that the pipeline's availability and performance stay consistent. You can keep updating the pipeline without interrupting the service.
You can publish one pipeline version to a service at a time. To publish multiple pipelines through Gateway, create one service for each pipeline. You can also publish the same pipeline to multiple services. This is useful if you want different service levels for different use cases, or if you want to publish different pipeline versions at once.
You publish a pipeline to a service from the Builder, where you choose the version to publish and either select an existing service or create a new one.
Callers connect to a chat or generative service through either the native Platform API or the OpenAI-compatible Completions API. For steps, see Call the Service from Your App.
You can also create a service without a pipeline and publish a local Haystack pipeline or agent to it later with the CLI or a coding agent. For details, see Publish a Pipeline to a Service.
Main Benefits
- Stable production endpoints: Callers connect to a service, not a pipeline. The pipeline can keep changing underneath without affecting what is live.
- Safe iteration: Experiment freely on your pipeline without touching what your users depend on.
- Version control with intent: Promote a pipeline version to a service when it is ready, instead of relying on auto-generated history.
- One pipeline, multiple services: Dev, staging, and production each independently managed from the same pipeline.
Publishing to a Service vs. Deploying a Pipeline
You can make a pipeline available in two ways: by publishing it to a service, or by deploying it without a service:
-
Deploying a pipeline makes it available for queries directly. It's best for quick testing and internal use, where endpoint stability is not a priority. When you update the pipeline and want to deploy a new version, the endpoint URL will change. Deploying a pipeline is required to test it in the Playground.
-
(Recommended) Publishing to a service assigns a specific pipeline version to a stable endpoint. The endpoint doesn't change when you update the pipeline. This is the recommended approach for production integrations, where your callers need a consistent endpoint they can rely on.
When building a pipeline other systems will connect to, use a service.
Code Builds
A code build is a service revision that runs code you wrote locally instead of a pipeline defined in YAML. Use a code build when your pipeline, agent, or function is easier to express in Python than in a pipeline configuration. Code builds are available only for serverless services of organizations that run on the Haystack fleet.
Code Bundles
A code bundle is a tar.gz archive of your code. Haystack Enterprise Platform stores each bundle in your organization's workspace, identified by the SHA-256 hash of its contents. Because bundles are content-addressed, uploading the same code twice stores it once, and a revision always points to exactly the code you uploaded.
You upload a bundle with the code bundles API as a multipart upload. You can also look up a bundle by its SHA-256 hash.
Code Revisions
When you create a revision for a service, you can reference an uploaded bundle with the code_bundle field instead of a pipeline source. The field has these properties:
sha256: The SHA-256 hash of the uploaded bundle, as 64 lowercase hexadecimal characters.entrypoint: The object Haystack Enterprise Platform imports from the bundle, in the formatmodule.py:attribute, for exampleapp.py:app.kind: What the entrypoint is:pipeline,agent,super_component, orfunction.input_schema: Optional. The JSON schema of the entrypoint's input, computed at deploy time. For function code builds, Haystack Enterprise Platform uses it to validate payloads.graph: Optional. A read-only graph of the pipeline for display, the output ofPipeline.to_dict().
In a code revision, config_yaml holds only settings, such as dependencies, inputs, and outputs. The pipeline itself comes from the bundle's entrypoint.
To check whether a revision runs a pipeline defined in YAML or a code build, look at the artifact_type field in the revision response. It's yaml for a revision that runs a pipeline defined in config_yaml, and code_bundle for a code build. For code builds, the source_metadata.code_bundle field of the revision holds the bundle's sha256, entrypoint, kind, size_bytes, input_schema, and graph.
Calling a Code Build
How you call a code build depends on its kind:
- For
pipeline,agent, andsuper_componentbuilds, use the same search and chat endpoints that you use for other deployments. - For
functionbuilds, use therunendpoint. Send any JSON object as the request body. Haystack Enterprise Platform passes it to your function as its payload and returns what the function returned in aresultfield.
For an example request, see Run a Function Code Build.
To learn how code builds run on isolated workers and which limits apply, see Code Builds.
Service Metrics
Clicking a service in Gateway opens the Service Overview page where you can review service performance and history. You can switch between the sections to check how to connect to the service, see its traces
On the Metrics tab, the CPU and memory charts show a Limit line, so you can see how close the service is to using all of its compute. If the service has a configured limit, the charts use it. The charts show no Limit line when no limit data is available yet, for example for a completely idle service. The replica chart shows Min and Max lines from the service's hosting settings.
If a service runs in a fleet environment, its CPU, memory, and replica metrics on the Metrics tab include load from all services that share the environment's instances, not just this one. To see how much each service uses, check the environment's metrics in Organization settings > Environments. For details, see View Environment Metrics. To see the environment's combined CPU, memory, and replica usage, see Check an Environment's Resource Usage.
Service Traces
You can view the traces for a service on the Service Traces page. You can filter the traces status, source, latency, and more. For details, see Traces and Trace Your Pipeline.
Hosting Settings
When you open a service and switch to the Settings tab, you land on a page where you can review and update all service settings, including hosting.
You can configure the hosting settings for a service to define how scalable and reliable it is. This includes:
- Minimum and maximum number of replicas available. Replicas are duplicate versions of a pipeline that are always available to handle traffic. When the traffic volume increases, Haystack Platform seamlessly switches to a functioning replica to maintain uninterrupted service.
- Idle timeout. The time after which a service enters standby mode to save resources. A pipeline gets automatically activated if you start searching with it, or you can activate it manually on the Pipelines page. When a pipeline is idle, your data remains perfectly safe and accessible anytime.
- Maximum concurrency per replica. The maximum number of requests one replica of the service handles at the same time. Use this setting to limit how many requests fail when one request goes wrong, not to increase throughput. One replica can serve requests from different callers of the same service, so a single request that exhausts the replica's memory can take down every other request that replica is currently serving. Leave this setting empty to use the platform default. Set it to 1 or 2 for a service whose requests are heavy or whose payloads are large, for example one that processes images or long documents. This setting takes effect only on services that run on serverless hosting. On other services, the value is saved but not applied. You set it with the API after you've created the service, not when you create it. For steps, see Limit Requests per Replica with the REST API.
- Instance size. The compute size given to each replica: Small, Medium, Large, or Extra large. Medium is the default. You choose it when you create a service, and you can change it later for services on serverless hosting. Instance size controls how much compute each replica gets, while the replica settings above control how many replicas run. Choose a larger size if each replica needs more compute, or a smaller one to save resources. Changing the instance size restarts the service's replicas on the new size; any queries already running finish on the old replicas first. To change it, see Manage a Service.
For production and business-critical use cases, we recommend the default hosting settings: 2 replicas at minimum and 24 hours idle timeout. For testing and low demand pipelines, we recommend 0 minimum replicas and 20 minutes idle timeout to save resources.
Instead of configuring hosting individually, a service can inherit its hosting settings from a shared fleet environment. Services in the same environment share infrastructure, which helps you manage cost and resource allocation. For how environments work, see Environments. For setup steps, see Manage Fleet Environments.
Service and Pipeline Hosting Settings
Hosting settings for a service are separate from the pipeline's hosting settings. When you publish a pipeline to a service, it uses the service's hosting settings. When you deploy a pipeline without a service, it uses the pipeline's hosting settings configured in pipeline settings. For details, see Pipeline Hosting Settings.
Service Secrets
A service can have its own secrets, separate from workspace and organization secrets. If a service secret has the same name as a workspace or organization secret, the service uses its own. This lets you give one service its own credentials, for example a production API key, without changing what other services in the workspace use.
You add and manage service secrets on the Secrets tab of the service details page. For steps, see Add Secrets to a Service.
Environments
An environment is a shared set of compute instances that services run on. Services in the same environment share one set of instances, so their requests add up to the environment's load. Services that don't use an environment run isolated.
An environment defines the size, the number of replicas, and the idle timeout for every service in it. Organization Admins create and manage environments on the Environments tab in Organization Settings. For each service, you choose under Hosting in the service's settings whether it uses an environment or its own custom settings.
Because the instances are shared, the CPU, memory, and replicas you see for a service also reflect the load of the other services in its environment. When you move a service to another environment, or switch it to custom settings, the service restarts. Changing an environment's size restarts every service in it.
To check how much load an environment carries, including its combined CPU, memory, and replica usage, see View Environment Metrics and Check an Environment's Resource Usage.
To create, edit, or delete environments, see Manage Fleet Environments. To choose an environment or custom hosting for a service, see Create a Service and Manage a Service.
If your organization admin has set up fleet environments, you can assign a service to one from the service hosting settings. Services assigned to the same fleet environment share underlying infrastructure, which helps manage cost and resource allocation. By default, a service is set to Own (isolated), meaning it doesn't share infrastructure with other services. Changing a service's environment restarts the service.
Organization admins create and manage fleet environments from Organization Settings. For details, see Managing Organization Settings.
Service Status
Possible statuses of a service are:
- Active — the service is deployed and ready to handle requests.
- Standby — the service is idle to save resources. It reactivates automatically when a request arrives, or you can reactivate it manually. For details, see Manage a Service.
- Deploying — the service is being deployed. On a first publish, nothing is serving yet. When you publish a new version, the version already serving keeps handling traffic until the rollout finishes. If the rollout fails, the last working version keeps serving, so callers don't notice. To see which pipeline version is rolling out, open the pipeline's versions list in Builder. The version shows a Publishing to 1 service tag. For details, see Manage Pipeline Versions.
- Failed — the service could not be deployed or encountered a fatal error.
- Inactive — the service has been deactivated.
You can deactivate a service from the Service Details page. To activate it again, use Activate service on that page.
AI Operator
To lower what a service costs without losing answer quality, attach an AI Operator to it. An AI Operator is an optimization goal for one service: it measures what the service costs and how well it answers, and then suggests cheaper configurations that keep the quality you need. For details, see AI Operator. To add one, see Set Up an AI Operator for a Service.
Durable Runs
A durable run is an alternative way to submit a request to a service. Instead of waiting on an open connection for the pipeline to finish, you submit the request, get a receipt immediately, and the pipeline keeps working even if you close the browser, lose your connection, or the underlying worker restarts. You come back later, or watch live, to see checkpoint progress and the final result.
Use durable runs for long-running pipelines, so you don't have to hold a connection open for the whole run.
Durable runs are only available for a service when:
- The organization has the durable runs capability turned on.
- The pipeline published to the service supports durable execution.
If either condition isn't met, the durable run controls are hidden and any in-progress observation stops, but accepted work keeps running.
Permissions
To see the Runs tab, open run deep links, and connect in durable mode, you need the search-history: read permission, the same one you need for service deployment history. To cancel a durable run, you also need the services: write permission.
Submission and Idempotency
When you submit a durable run, the request includes a stable Idempotency-Key. If you retry the exact same request, the service recognizes the key and avoids starting a duplicate execution. If the service is at capacity, you see a message saying capacity is full, and you can retry later. When a submission is rejected because the service is at capacity or the request was invalid, your next submission uses a new idempotency key, so it counts as a new request, not a duplicate.
For steps to view and manage durable runs, see Manage a Service.
Related Information
Was this page helpful?