Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Pipeline Settings and Scaling

You can adjust the number of replicas available to a pipeline and its idle timeout to save costs and meet your traffic requirements. You configure these settings, together with the MCP tool, GPU support, and feedback tags, on the pipeline's Settings tab, where you can also check the pipeline ID.

Pipeline Hosting Settings​

You can configure the following hosting settings:

  • Minimum and maximum number of replicas. Replicas are the number of duplicate versions of a pipeline that are available. If demand spikes, Haystack Platform switches to a functioning replica to keep the service running. The minimum is the lowest number of replicas running while the pipeline has traffic. The maximum is the highest number the platform adds, and it can't exceed 10.
  • Idle timeout. Idle timeout is the time after which an unused pipeline enters standby mode and Haystack Platform scales its replicas down to zero, minimum replicas included. The maximum idle timeout is 30 days. Idle pipelines don't use up the pipeline hours included in your plan. For example, if you set the idle timeout to 12 hours, the pipeline enters standby mode after 12 hours of inactivity.

You can configure both settings in the pipeline's Settings tab or through the Update Pipeline REST API endpoint. To use a pipeline on standby, activate it either on the Pipelines page or by initiating a search using that pipeline.

You can use the Production toggle to set the pipeline hosting settings to recommended values, or you can configure your own settings.

Pipeline and Service Hosting Settings​

Hosting settings for a pipeline are separate from the service's hosting settings. When you publish a pipeline to a service, it uses the service's hosting settings. When you deploy a pipeline without a service, it uses the pipeline's hosting settings configured in pipeline settings. For details, see Gateway.

Pipeline Scaling​

To increase availability, set the number of replicas allocated to a pipeline. Haystack Platform automatically scales replicas up or down based on traffic and on the minimum and maximum you set. Adding more replicas can increase your costs.

How Scaling Works​

Each deployed pipeline has a pool of replicas that handle incoming requests. Scale-up and scale-down are both driven by request queue depth, which means the number of requests waiting to be processed relative to the current replica capacity. The platform stays within your configured minimum and maximum at all times.

Scaling Up​

When requests arrive faster than the current replicas can process them, the platform adds replicas up to the configured maximum. New replicas start accepting requests as soon as they're initialized. You don't need to take any action to trigger this. The platform detects queue depth and responds within seconds.

The number of replicas added at once scales with the size of the request queue. A sudden large spike brings up more replicas than a small gradual increase.

Scaling Down​

When the queue clears and request volume drops, the platform removes replicas down to the configured minimum. This keeps costs proportional to actual usage.

When a pipeline receives no requests for the entire idle timeout period, the platform removes all remaining replicas, including the minimum replicas. A subsequent request activates the pipeline: the first request waits while a replica initializes, and then processing continues as normal.

Cold Starts and Timeouts​

The first request to a pipeline or service that scaled to zero is slower than the following ones. This is a cold start. Before the request runs, the platform assigns a replica and prepares it: it installs the pipeline's dependencies, such as the Haystack version, custom components, and models. A cold start takes longer the more the pipeline needs to install, and it can exceed the startup timeouts below. Requests that follow use the warm replica and run at normal speed.

If the replica isn't ready in time, the first request fails. For 504 and 503, send the request again. For 408, retrying won't help:

  • 504 Gateway Timeout: The platform couldn't get a replica for the request within 120 seconds. This is a startup timeout. It doesn't mean your pipeline took too long to run.
  • 503 Service Unavailable: No replica of the right size was free within 100 seconds.
  • 408 Request Timeout: The pipeline started but didn't finish within the request timeout of 120 seconds. This is the pipeline timeout. For long runs, use the asynchronous or streaming endpoints instead. For details, see Timeouts.

To avoid cold starts before a demo or a known traffic peak, send a test request a few minutes ahead and set an idle timeout longer than the expected gap between requests. Minimum replicas alone don't prevent this.

Recommendations​

For testing and development pipelines, set minimum replicas to 0 and idle timeout to a short period, like 20 minutes. This minimizes cost for pipelines that are not always in use.

For production pipelines used in business-critical scenarios, set minimum replicas to 2 and maximum to 10, with a longer idle timeout, like 1 day. Running at least two replicas keeps your pipeline available even if one replica becomes unavailable. A long idle timeout keeps the replicas warm between requests, so fewer requests hit a cold start.

GPU Settings​

Pipelines use CPU by default. Turn on GPU support on the Settings tab when your pipeline includes components that run better on a GPU, such as embedders, generators, or custom components that use AI models.

When GPU support is on, the pipeline checks whether a component needs a GPU and assigns one automatically. GPUs are only used when required and are not reserved for the entire pipeline run.

If GPU support is off and your pipeline includes components that rely on a GPU, those components run on the CPU instead. This can slow down processing and may cause timeouts, especially for larger or more complex pipelines.

MCP Tool and Feedback Tags​

Use the MCP tool section to expose your pipeline as a Model Context Protocol (MCP) tool to AI coding assistants like Cursor, Claude Code, and GitHub Copilot. For details, see Use Your Pipelines as MCP Tools.

Use feedback tags to add tags that users can select when giving feedback on the pipeline's responses. This helps you group feedback and analyze it. For details, see Collect Feedback.