Limit Requests per Replica with the REST APIBeta
Cap how many requests one replica of a service handles at the same time.
About This Task
If a service on serverless hosting handles heavy or large requests, such as images or long documents, cap how many requests one replica serves at the same time. One request that exhausts a replica's memory then doesn't fail every other request on that replica.
You can set this only with the API, and only after you create the service. The API calls a service a deployment. On services that don't run on serverless hosting, the value is saved but has no effect.
Prerequisites
- You need a Haystack Platform API key. For details, see Generate API Keys.
- You need a service. For details, see Create a Service.
Set the Limit
Send a PATCH request to Patch Deployment. Replace <YOUR_WORKSPACE> with your workspace name, <DEPLOYMENT_ID> with the service's ID, and <YOUR_API_KEY> with your API key:
curl --request PATCH \
--url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/deployments/<DEPLOYMENT_ID> \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header 'authorization: Bearer <YOUR_API_KEY>' \
--data '
{
"max_concurrency_per_replica": 2
}
'
Check that the response returns the value you set. The change applies to the replicas that are already running.
To go back to the platform default, send the same request with max_concurrency_per_replica set to null.
What To Do Next
- Configure Service Hosting and Cost to change replicas, instance size, and idle timeout.
Related Information
Was this page helpful?