> For the complete documentation index, see [llms.txt](https://docs.cloud.deepset.ai/llms.txt)

Last modified: 2026-10-27

# Pipeline EvaluationBeta

> **Beta**
> 
> This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Evaluation turns "the answers seem fine" into numbers you can track. You decide what good looks like for your use case and write it down as a metric. Then you measure your pipeline against it on conversations it already had.

* * *

## What You Evaluate

You evaluate the conversations and answers your pipeline already produced. This lets you see how your pipeline behaves on real conversations, without having to prepare a test dataset first. To compare two pipeline versions on the same questions, you can also score them on a Dataset of test cases. See [Candidate and Baseline Experiments](https://docs.cloud.deepset.ai/docs/evaluation-experiments.md#candidate-and-baseline-experiments).

A *turn* is one pipeline run: one query and everything the pipeline did to answer it, including retrieved documents, tool calls, and the final reply. The platform records each turn as a *trace*.

A *session* is a conversation: all the turns that share one chat. A single query sent outside a chat counts as a session with one turn. In the Evaluation screens, Traces, and Search History, the platform calls a session a *conversation*, and this page uses both terms.

Every query you send to your pipeline, from Playground, a deployed app, or the API, becomes a session you can evaluate. Your evaluation reflects how people use the pipeline, not how you expect them to use it.

You evaluate whole sessions, so a metric can judge things a single answer can't show, such as whether the pipeline remembered earlier questions or whether an agent called the right tool. An LLM judge can also see which tools and skills the agent was offered in each turn, so it can tell when the agent missed a better one.

## Main Benefits

Evaluation lets you:

-   Check whether a pipeline is ready for production by measuring it on a realistic set of conversations before you share it widely.
-   See whether a change helped by measuring before and after you change a prompt, a model, or a retrieval setting.
-   Find where the pipeline fails by inspecting the sessions with the lowest scores instead of reading every conversation.
-   Catch problems that nobody reports by measuring production conversations.

User feedback answers some of these questions too, but it depends on people taking the time to rate answers. Metrics run on every session you choose, so you get consistent coverage.

## Where You Find Evaluation

You evaluate a pipeline on its own Evaluation tab: open a pipeline and click **Evaluation**. The tab shows only the content that belongs to that pipeline, including its Evaluation goals. See [Evaluation Goals](https://docs.cloud.deepset.ai/docs/evaluation-goals.md).

There's no workspace-wide Evaluation page. To evaluate, pick a pipeline from the pipelines list. From a Service, click **Evaluate** to open the Evaluation tab of its source pipeline.

If you used Replay Suites before, use Experiments instead: old links to Replay Suites now open the Experiments page. See [Evaluation Experiments](https://docs.cloud.deepset.ai/docs/evaluation-experiments.md).

## How Evaluation Fits Together

Evaluation has four building blocks. Each has its own page:

-   *Metrics*: The questions you ask about every session, answered by an LLM judge or by code. See [Evaluation Metrics](https://docs.cloud.deepset.ai/docs/evaluation-metrics.md).
-   *Datasets*: Collections of test cases you keep for measuring pipelines. See [Evaluation Datasets and Metric Groups](https://docs.cloud.deepset.ai/docs/evaluation-datasets-and-metric-groups.md).
-   *Metric Groups*: Named sets of metrics, so you can pick related metrics together instead of one by one. See [Evaluation Datasets and Metric Groups](https://docs.cloud.deepset.ai/docs/evaluation-datasets-and-metric-groups.md#metric-groups).
-   *Experiments*: Runs of a set of metrics over your sessions that you repeat to see whether a change helped, including Candidate and Baseline comparisons. See [Evaluation Experiments](https://docs.cloud.deepset.ai/docs/evaluation-experiments.md).

## A Measurement, Not a Verdict

Results tell you what happened, not whether it's good enough. There is no built-in pass or fail. You decide what threshold matters for your use case: 0.7 groundedness may be fine for an internal assistant and not good enough for a customer-facing one.

## Related Information

-   [Evaluation Metrics](https://docs.cloud.deepset.ai/docs/evaluation-metrics.md)
-   [Evaluation Datasets and Metric Groups](https://docs.cloud.deepset.ai/docs/evaluation-datasets-and-metric-groups.md)
-   [Evaluation Experiments](https://docs.cloud.deepset.ai/docs/evaluation-experiments.md)
-   [Write and Test an Evaluation Metric](https://docs.cloud.deepset.ai/docs/write-and-test-evaluation-metrics.md)
-   [Run an Experiment](https://docs.cloud.deepset.ai/docs/run-an-experiment.md)
-   [Evaluating Step-by-Step](https://docs.cloud.deepset.ai/docs/evaluating-your-pipeline.md)
-   [Trace with Built-In Traces](https://docs.cloud.deepset.ai/docs/trace-your-pipelines-ui.md)
-   [Collect User Feedback](https://docs.cloud.deepset.ai/docs/collect-feedback.md)
