Tune CPU Embedders

If your query pipeline runs an embedder such as SentenceTransformersTextEmbedder on CPU and handles several requests at the same time, you can limit the number of threads the model uses. This improves throughput and reduces tail latency.

---

About This Task

A pipeline that runs a Sentence Transformers embedder on CPU, without a GPU or a Triton model server, loads one model in each query pod and shares it across all parallel requests. Each embedding call starts a pool of native threads. By default, the size of that pool depends on the number of CPU cores the pod can see, not on the CPU the pod is allowed to use. When many requests run at once, their thread pools compete for the same CPU, and queries become slower. In some cases, requests fail with WorkerError.

To avoid this, Haystack Enterprise Platform sets default thread limits in query pods, and you can override them. How you do that depends on the backend:

• Torch (the default backend) reads the thread limits from environment variables.
• ONNX ignores these environment variables. You set the thread count in the embedder's model_kwargs instead.

Check the Default Thread Limits

In query pods, Haystack Enterprise Platform sets these environment variables:

Variable	Default value
OMP_NUM_THREADS	The pod's CPU request, rounded up to a whole core, and at least 1. For example, a request of 500m becomes 1 and a request of 1,730m becomes 2.
MKL_NUM_THREADS	The same value as OMP_NUM_THREADS.
TOKENIZERS_PARALLELISM	false

The defaults apply to query pods only, not to indexing.

Override the Defaults for Torch

Use this section if your embedder uses the default Torch backend. To use different values, add a workspace secret with the same name as the variable. A secret takes precedence over the default.

1. Add a secret named OMP_NUM_THREADS with the number of threads you want, for example 1. For instructions, see Add Secrets.
2. Repeat for MKL_NUM_THREADS and, if needed, TOKENIZERS_PARALLELISM.
3. Redeploy your pipeline so the query pods pick up the new values. If throughput doesn't change, check that the secret names match the variable names exactly and that you redeployed the pipeline after adding them.

Set the Thread Count for ONNX

To apply the change, edit the embedder in your query pipeline YAML and redeploy the pipeline.

ONNX Runtime ignores OMP_NUM_THREADS and MKL_NUM_THREADS. To limit its threads, set session_options in the embedder's model_kwargs:

  query_embedder:
    type: haystack_integrations.components.embedders.sentence_transformers.SentenceTransformersTextEmbedder
    init_parameters:
      model: intfloat/e5-base-v2
      backend: onnx
      model_kwargs:
        file_name: onnx/model.onnx
        session_options:
          intra_op_num_threads: 1

This setting requires sentence-transformers-haystack 2.1.0 or later. For OptimumTextEmbedder, it requires optimum-haystack 4.1.0 or later.

You can set only options that take plain values, such as thread counts and booleans. Options that take an enum value, such as graph_optimization_level, can't be set in YAML.

Choose a Thread Count

The best value depends on how many requests your pipeline handles at the same time. The following table shows embeddings per second from one local test with four parallel callers. It ran on linux/arm64 with Docker limited to two or four CPUs. Your results may differ.

Variant	2 CPUs	4 CPUs
Torch, default	3.28	8.24
Torch, thread environment variables set to 1	6.54	13.10
ONNX, default	2.52	6.81
ONNX, intra_op_num_threads: 1	12.54	25.61

Limiting the threads has a trade-off: a single caller gets slower with one thread because the model can no longer spread one request over several cores. Use one thread if your pipeline serves several requests in parallel. If it handles few concurrent requests, keep the defaults.

What To Do Next

• SentenceTransformersTextEmbedder
• Add Secrets