Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Service Hosting

Hosting settings define how scalable and reliable a service is. This page explains what each setting does and how services can share compute in an environment.


Hosting Settings​

When you open a service and switch to the Settings tab, you land on a page where you can review and update all service settings, including hosting.

You can configure the hosting settings for a service to define how scalable and reliable it is. This includes:

  • Minimum and maximum number of replicas available. Replicas are duplicate versions of a pipeline that handle traffic. The minimum keeps at least one replica ready, but a replica loads the pipeline when it receives its first request, so it can answer slowly at first. This happens after the first activation, a revision switch, or a top-up that replaces a replica. For details, see Why a Service Starts Slowly. When the traffic volume increases, Haystack Platform switches to a functioning replica to maintain uninterrupted service.
  • Idle timeout. The time after which a service enters standby mode to save resources. A pipeline gets automatically activated if you start searching with it, or you can activate it manually on the Pipelines page. When a pipeline is idle, your data remains perfectly safe and accessible anytime.
  • Maximum concurrency per replica. The maximum number of requests one replica of the service handles at the same time. Use this setting to limit how many requests fail when one request goes wrong, not to increase throughput. One replica can serve requests from different callers of the same service, so a single request that exhausts the replica's memory can take down every other request that replica is currently serving. Leave this setting empty to use the platform default. Set it to 1 or 2 for a service whose requests are heavy or whose payloads are large, for example one that processes images or long documents. This setting takes effect only on services that run on serverless hosting. On other services, the value is saved but not applied. You set it with the API after you've created the service, not when you create it. For steps, see Limit Requests per Replica with the REST API.

For production and business-critical use cases, we recommend the default hosting settings: 2 replicas at minimum and 24 hours idle timeout. For testing and low demand pipelines, we recommend 0 minimum replicas and 20 minutes idle timeout to save resources.

Service and Pipeline Hosting Settings​

Hosting settings for a service are separate from the pipeline's hosting settings. When you publish a pipeline to a service, it uses the service's hosting settings. When you deploy a pipeline without a service, it uses the pipeline's hosting settings configured in pipeline settings. For details, see Pipeline Hosting Settings.