Configure Service Hosting and Cost
Tune how much compute a service gets, how many replicas run, and how long the service stays active. These settings balance responsiveness against cost.
Prerequisites
- You must have read and write permissions to services. For details, see User Roles and Permissions.
- You must have a service. For details, see Create a Service. For a pipeline over 256 KB, see Run a Pipeline Larger Than 256 KB.
Scale the Service and Control Cost
These settings apply only to this service. If the same pipeline is also deployed on its own, configure that deployment separately in the pipeline deployment settings.
Credits consumed ≈ min replicas × hours deployed. Services on standby or in inactive status consume no credits.
- In Haystack Enterprise Platform, click Gateway in the navigation and open the service.
- Click Settings and go to the Hosting section.
-
For a service on serverless hosting, to give each replica more or less compute, select a size under Instance size. The list offers the sizes your organization has enabled: Small, Medium, Large, or Extra large. Organization admins choose these sizes in Organization Settings. For details, see Managing Compute Settings. If the service already runs on a size your organization has since turned off, that size stays in the list so you can see the current value. Confirm the change when prompted.
The service's replicas restart on the new size. Queries already running finish on the old replicas first. If your organization has no capacity for the size you chose, an error appears and the size stays the same. Contact your Organization Admin.
- Choose a preset, or select Custom to set your own values:
- High demand: For production traffic that needs high availability and scaling.
- Low demand: For testing or services that are used rarely.
- Custom: Set Min replicas, Max replicas, and Idle timeout yourself.
- Confirm your changes.
You can change these settings at any time. To check whether the service needs more resources before you change them, see Check Service Metrics.
To cap how many requests one replica serves at the same time, see Limit Requests per Replica with the REST API.
What Min Replicas Guarantees
If your service answers slowly right after it starts, this is why. Min replicas keeps that many replicas claimed and ready for your service. It guarantees at least one ready replica, not that every replica has already loaded your pipeline. A replica in the minimum set loads the pipeline when it receives its first request. A pipeline with a long preparation, for example one that loads a local embedding model, answers slowly on a replica that hasn't loaded it yet.
You can see this after:
- The first activation of a service.
- A revision switch, when you publish a new revision to the service.
- A top-up, when the platform starts a replacement to bring the service back up to Min replicas.
During this time, queries can take much longer than usual. A streaming response can send only ping events with phase set to cold_start. If many queries wait on replicas that are still loading, the service can reject further queries with a Too many pending requests for this version error. For details on cold_start pings, see Stream Pipeline Responses with the REST API.
Replicas in the minimum set consume credits while they are claimed, including before they load the pipeline.
After an activation or a revision switch, send a few test queries before real traffic arrives. Send about one query for each replica you expect, with the Try it section of the service or with the REST API. Wait until the queries answer in your normal response time, then start the load. If a query fails with Too many pending requests for this version, wait and send the test queries again.
Shorten the Cold Start
Warming hides a cold start, but a faster pipeline start removes it. To shorten the start:
- Avoid large dependencies. Every package under
dependenciesin the pipeline YAML, and every external dependency of a custom component, is installed when a replica prepares your pipeline. Large packages, such astorch,transformers, orsentence-transformers, can add minutes. Use the components that come with Haystack Platform where you can, and add only the packages your pipeline needs. - Don't load models locally. A component that downloads and loads a model on the replica, such as a local embedder, ranker, or language model, repeats that work on every replica that starts. Use a hosted model instead: the model stays loaded on the provider's side, and the replica only makes an API call. For the options, see Using Hosted Models and External Services.
Load a model locally only if it's a hard requirement, for example because your data can't leave the platform, and only if you can afford the cold start. In that case, pick the smallest model that meets your needs and warm the service before every expected load.
After you change the pipeline and publish a new revision, send a test query and compare the response time with your normal response time. For help reading the results, see Check Service Metrics.
Set Max Concurrency per Replica
Another replica starts when each running replica has more than the Max concurrency per replica number of requests in flight. Lower the value for slow pipelines, such as agents or OCR, so the service scales out sooner. This setting is available for services on serverless hosting. It's hidden while the service uses a fleet environment.
- Open the service, click Settings, and go to the Hosting section.
- In Max concurrency per replica, type a whole number from 1 to 24. A replica handles at most 24 requests at once, so a higher value would never start another replica.
- Confirm the change when prompted, or click Cancel to discard it. After you confirm, the field shows your value instead of
8 (default).
The value saves on its own, separately from the High demand, Low demand, and Custom presets. Switching presets doesn't change it. Running queries aren't interrupted.
If you haven't set a value, the default is 8. Clearing the field doesn't reset a saved value. To go back to the default, enter 8.
Use a Fleet Environment or Custom Hosting
To give a service the same hosting as other services, connect it to a fleet environment. It then inherits the environment's size, replicas, and idle timeout instead of using its own. Only services on serverless hosting can use a fleet environment.
- Open the service, click Settings, and go to the Hosting section.
- Choose Environment and pick an environment from the list, or choose Custom to set the service's own Min replicas, Max replicas, and Idle timeout.
- Confirm the restart when prompted.
With Environment, the environment's settings show read-only, with a Manage environments link to the Environments tab where you can change them. Switching between Environment and Custom, or between two environments, restarts the service. If you switch to Custom, the service restarts with its own saved settings.
Services in the same environment share instances, so their requests add up to the environment's load. When you check CPU and memory on the Metrics tab for a service that shares an environment, the charts show usage for every service in that environment, not only this one. Their titles end with "(shared across the environment)".
Only an organization admin can create, rename, or delete fleet environments. To check the load on a shared environment, see View Environment Metrics and Check an Environment's Resource Usage. For details on creating and editing environments, see Manage Fleet Environments.
Keep a Service Active with Scheduled Calls
Turn on scheduled calls to make the service call itself at regular intervals, so it doesn't go idle between real requests. This helps if your service gets sporadic traffic and you still want fast response times. Scheduled calls work only for services on non-serverless hosting.
- Open the service, click Settings, and go to the Scheduled Calls section.
- Turn on Enabled.
- In Every (minutes), set how often the service should be called.
- Optional: In Query, type the text to send with each scheduled call. Leave it empty to dispatch an empty query.
- Optional: In Params (JSON), type extra parameters to send as valid JSON.
- Click Save.
If a schedule already exists, the section shows Last call, which is the time of the most recent scheduled call. Turn off Enabled and save to stop scheduled calls without losing your saved interval, query, and parameters.
To run a long scheduled job, such as a nightly flow, turn on Run as a durable execution before you click Save. Each scheduled call is then submitted as a durable run that isn't cut off at the request time limit and is retried if a worker restarts. For details, see Run Scheduled Calls as Durable Runs.
Run a Pipeline Larger Than 256 KB
Saving a pipeline or index configuration larger than 256 KB fails with error 413. This limit applies when you create, validate, or update a pipeline, pipeline version, or index, in Builder or through the API. The limit doesn't apply to a YAML configuration pushed to a service as a revision, so run a pipeline over 256 KB as a serverless service. You push the YAML with the REST API, and you can't edit the pipeline in Builder afterwards. For the steps, see Run a Pipeline Larger Than 256 KB with the REST API.
What To Do Next
- Monitor and Debug a Service to see how the service behaves under load.
- Deactivate or Delete a Service to deactivate, wake, or delete the service.
Related Information
Was this page helpful?