Limit Requests per Replica with the REST API
Cap how many requests one replica of a service handles at the same time.
About This Task
If a service on serverless hosting handles heavy or large requests, such as images or long documents, cap how many requests one replica serves at the same time. One request that exhausts a replica's memory then doesn't fail every other request on that replica.
Prerequisites
- You need a Haystack Platform API key. For details, see Generate API Keys.
- You need a service. For details, see Create a Service.
Set the Limit
Send a PATCH request to Patch Deployment. Replace <YOUR_WORKSPACE> with your workspace name, <DEPLOYMENT_ID> with the service's ID, and <YOUR_API_KEY> with your API key:
curl --request PATCH \
--url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/deployments/<DEPLOYMENT_ID> \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header 'authorization: Bearer <YOUR_API_KEY>' \
--data '
{
"max_concurrency_per_replica": 2
}
'
Check that the response returns the value you set. The change applies to the replicas that are already running.
The value must be between 1 and 24. The API ignores null in a PATCH request, so sending null doesn't reset the setting. To go back to the platform default, set max_concurrency_per_replica to 8.
What To Do Next
- Configure Service Hosting and Cost to change replicas, instance size, and idle timeout.
Related Information
Was this page helpful?