Service Hosting
Hosting settings define how scalable and reliable a service is. This page explains what each setting does and how services can share compute in an environment.
Hosting Settings
When you open a service and switch to the Settings tab, you land on a page where you can review and update all service settings, including hosting.
You can configure the hosting settings for a service to define how scalable and reliable it is. This includes:
- Minimum and maximum number of replicas available. Replicas are duplicate versions of a pipeline that handle traffic. The minimum keeps at least one replica ready, but a replica loads the pipeline when it receives its first request, so it can answer slowly at first. This happens after the first activation, a revision switch, or a top-up that replaces a replica. For details, see Why a Service Starts Slowly. When the traffic volume increases, Haystack Platform switches to a functioning replica to maintain uninterrupted service.
- Idle timeout. The time after which a service enters standby mode to save resources. A pipeline gets automatically activated if you start searching with it, or you can activate it manually on the Pipelines page. When a pipeline is idle, your data remains perfectly safe and accessible anytime.
- Maximum concurrency per replica. The maximum number of requests one replica of the service handles at the same time. Use this setting to limit how many requests fail when one request goes wrong, not to increase throughput. One replica can serve requests from different callers of the same service, so a single request that exhausts the replica's memory can take down every other request that replica is currently serving. Leave this setting empty to use the platform default. Set it to 1 or 2 for a service whose requests are heavy or whose payloads are large, for example one that processes images or long documents. This setting takes effect only on services that run on serverless hosting. On other services, the value is saved but not applied. You set it with the API after you've created the service, not when you create it. For steps, see Limit Requests per Replica with the REST API.
- Instance size. The compute size given to each replica: Small, Medium, Large, or Extra large. Medium is the default. You choose it when you create a service, and you can change it later for services on serverless hosting. Instance size controls how much compute each replica gets, while the replica settings above control how many replicas run. Choose a larger size if each replica needs more compute, or a smaller one to save resources. Changing the instance size restarts the service's replicas on the new size; any queries already running finish on the old replicas first. To change it, see Manage a Service.
For production and business-critical use cases, we recommend the default hosting settings: 2 replicas at minimum and 24 hours idle timeout. For testing and low demand pipelines, we recommend 0 minimum replicas and 20 minutes idle timeout to save resources.
Instead of configuring hosting individually, a service can inherit its hosting settings from a shared fleet environment. Services in the same environment share infrastructure, which helps you manage cost and resource allocation. For how environments work, see Environments. For setup steps, see Manage Fleet Environments.
Service and Pipeline Hosting Settings
Hosting settings for a service are separate from the pipeline's hosting settings. When you publish a pipeline to a service, it uses the service's hosting settings. When you deploy a pipeline without a service, it uses the pipeline's hosting settings configured in pipeline settings. For details, see Pipeline Hosting Settings.
Environments
An environment is a shared set of compute instances that services run on. Services in the same environment share one set of instances, so their requests add up to the environment's load. Services that don't use an environment run isolated.
An environment defines the size, the number of replicas, and the idle timeout for every service in it. Organization Admins create and manage environments on the Environments tab in Organization Settings. For each service, you choose under Hosting in the service's settings whether it uses an environment or its own custom settings.
Because the instances are shared, the CPU, memory, and replicas you see for a service also reflect the load of the other services in its environment. When you move a service to another environment, or switch it to custom settings, the service restarts. Changing an environment's size restarts every service in it.
To check how much load an environment carries, including its combined CPU, memory, and replica usage, see View Environment Metrics and Check an Environment's Resource Usage.
To create, edit, or delete environments, see Manage Fleet Environments. To choose an environment or custom hosting for a service, see Create a Service and Manage a Service.
By default, a service is set to Own (isolated), meaning it doesn't share infrastructure with other services.
Related Information
Was this page helpful?