> For the complete documentation index, see [llms.txt](https://docs.cloud.deepset.ai/llms.txt)

Last modified: 2026-11-03

# Evaluating Step-by-Step

## Why Should I Evaluate?

Before bringing your pipeline to production, you want to make sure it's useful, provides relevant results, and serves the intended purpose. By evaluating your app, you can determine how effective it is and how satisfied the users are. You can share your pipeline prototype with external users and collect their feedback in Haystack Platform interface. Prototypes are customizable, you can add your brand logo and colors to them.

* * *

User feedback tells you how people feel about the answers, but only for the answers they rate. To measure every conversation consistently, and to check whether a change made your pipeline better or worse, use metrics and experiments. A metric answers one question about each session, such as "Was the answer grounded in the documents?" An experiment runs your metrics over the sessions you choose and shows you where the pipeline does well and where it struggles. For detailed steps, see [Build an Evaluation Dataset](https://docs.cloud.deepset.ai/docs/build-an-evaluation-dataset.md). For background, see [Pipeline Evaluation](https://docs.cloud.deepset.ai/docs/pipeline-evaluation.md).

The two approaches work best together: user feedback shows you what matters to your users, and metrics let you measure it at scale.

## Detailed Steps

Here’s a breakdown of the steps we recommend you take to evaluate your pipeline:

1.  [Upload files to your workspace](https://docs.cloud.deepset.ai/docs/upload-files.md). The pipeline you’re evaluating will run on these files. The more files you upload, the more difficult the task of finding the right answer is.
2.  [Create a pipeline](https://docs.cloud.deepset.ai/docs/create-a-pipeline-in-studio.md). You can start with one of the ready-made templates.
3.  [Share your pipeline with users](https://docs.cloud.deepset.ai/docs/share-a-pipeline-prototype.md). At this stage, you want to get an idea of how people use your pipeline and if it works OK. You’re not aiming for high user satisfaction at this point. You just want to determine if it’s ready for an evaluation through experiments. If most of your users are happy with the results, it means it’s good to go. Otherwise, try tweaking your app until its performance is satisfactory.
4.  [Collect user feedback](https://docs.cloud.deepset.ai/docs/collect-feedback.md). It's best to have domain experts and qualified users try your pipeline and then use their feedback to improve it.
5.  [Optimize and improve your pipeline](https://docs.cloud.deepset.ai/docs/improving-your-qa-pipeline.md).

## Measure Your Pipeline With Metrics and Experiments

Once your pipeline works and people use it, measure it. These steps help you find out how well it performs, where it fails, and whether your changes help.

You can work from the workspace or from a single pipeline. To see everything in the workspace, click **Evaluation** under *Testing* in the sidebar. To focus on one pipeline, open it and click **Evaluation**. Both open the same pages, and the pipeline tab shows only that pipeline's content.

To work on a pipeline, open it and click **Evaluation**.

1.  Collect sessions to measure. Every query sent to your pipeline creates a session: from Playground, a shared prototype, a deployed app, or the API. If your pipeline isn't in use yet, run a set of queries that represent what your users ask, including difficult ones.
2.  Decide what good looks like. Write down two or three questions that matter most for your use case, for example "Is the answer grounded in the documents?" and "Did the agent search before answering?" Use what you learned from user feedback to choose them.
3.  Turn each question into a metric and test it. Use an LLM judge for qualities that need judgment, and code for things you can check exactly. Test each metric on sessions where you already know the right answer until you trust its results. Each new metric joins at least one Metric Group. See [Write and Test an Evaluation Metric](https://docs.cloud.deepset.ai/docs/write-and-test-evaluation-metrics.md).
4.  Create an experiment and run it. Group your metrics into an experiment and run it on the sessions you want to measure. See [Run an Experiment](https://docs.cloud.deepset.ai/docs/run-an-experiment.md).

To get help setting up the experiment, click **Create Experiment with AI** on the Experiments page. See [Create an Experiment](https://docs.cloud.deepset.ai/docs/run-an-experiment.md#create-an-experiment).

5.  Find the weak spots. Look at the distribution for each metric, then open the lowest-scoring sessions and read the rationales. They tell you whether the problem is in your pipeline or in your metric.
6.  Improve and measure again. Change your pipeline, send the same or similar queries to it, and run the experiment on the new sessions. Compare the results with the previous run to see whether the change helped.

To get the assistant's summary of the comparison, see [Get an AI Summary of an Experiment](https://docs.cloud.deepset.ai/docs/run-an-experiment.md#get-an-ai-summary-of-an-experiment).

Repeat steps five and six as your pipeline evolves. You can also run your experiment on recent production sessions to check that your pipeline still behaves as expected. To run the same test cases on a pipeline in another workspace, see [Import and Export a Dataset](https://docs.cloud.deepset.ai/docs/import-and-export-a-dataset.md).

## Check Whether Your Pipeline Meets Its Evaluation Goals

To find out whether your pipeline meets its Evaluation goals on real traffic right now, evaluate the goals on the pipeline's live Sessions. Each goal is a pass rule on a Metric, for example "at least 95% of Sessions pass".

### Evaluate Your Goals Now

Before you start, make sure the pipeline has at least one Evaluation goal and has received queries in the last 7 days. To add goals, see [Define Evaluation Goals on a Pipeline](https://docs.cloud.deepset.ai/docs/define-evaluation-goals.md).

1.  Open your pipeline and click **Evaluation**.
2.  In the pipeline's Evaluation goals, click **Evaluate now**.

**Result**: The platform scores the pipeline's live Sessions from the last 7 days with the Evaluators behind your goals. Each goal then shows its verdict. The verdicts appear when scoring finishes; if you leave the page, open the Evaluation goals again to see them. If no verdicts appear, see [Fix Evaluation Problems](https://docs.cloud.deepset.ai/docs/evaluating-your-pipeline.md#fix-evaluation-problems).

What the run does:

-   It scores only live Sessions from the last 7 days. Sessions from test runs aren't included.
-   It scores the newest Sessions first. A single run judges up to 1500 judge calls divided by the number of Evaluators (at most 500 Sessions). If the last 7 days have more Sessions, the run covers only the newest ones.
-   It uses a goal Metric Group that the platform manages for the pipeline. You can't use this group for a Deployment Gate or an optimization goal.
-   Each click can cost up to that many judge calls, because LLM judges call a model for every result. If a run for the pipeline's goals is already in progress when you click, the platform shows that run and doesn't start another, so you don't pay twice.

### Read the Verdicts

Each goal shows one of these verdicts:

-   *Met*: The share of scored Sessions that pass the goal's rule reaches the goal's required pass rate.
-   *Missed*: The share is below the required pass rate.
-   *No data*: The run has no Session it can judge for this goal, for example because the Metric didn't apply to any Session or the goal's rule doesn't fit the Metric's type.

A verdict includes the pass rate and how many Sessions it is based on, for example "92% of 140 Sessions pass, needs 95%". For a Score Metric, a Session passes when its mean score reaches the goal's threshold in the direction that is better for the Metric. For a Label Metric, a Session passes when all of its labels are among the goal's passing values.

If you change a goal's threshold or required pass rate, the verdict updates right away from the same run, and you don't need to evaluate again. A verdict is stale when the last run can no longer answer the goal as it is now: the run has no results for the goal's Metric, or the goal's Evaluator has a newer version. To get a current verdict, click **Evaluate now** again.

### Fix Evaluation Problems

-   *The pipeline has no live Sessions in the last 7 days*: Send queries to the deployed pipeline, for example from an app or the API, and evaluate again.
-   *The pipeline has no Evaluation goals*: Add at least one goal, and evaluate again.
-   *Too many Scoring Runs are in progress in your workspace*: Wait for some to finish, and evaluate again.
-   *The run couldn't be queued*: Try again in a moment. If it keeps failing, contact support.

For general information on running Metrics over Sessions, see [Run an Experiment](https://docs.cloud.deepset.ai/docs/run-an-experiment.md).

## Keep Important Queries in a Dataset

To reuse the queries that matter to you when you measure a change, keep them in a Dataset. A Dataset belongs to the workspace, so one Dataset can hold test cases from several pipelines. Each test case shows the pipeline it comes from, so you always know which pipeline produced it.

To add test cases from Search History, copy them between Datasets, or edit them, see [Build an Evaluation Dataset](https://docs.cloud.deepset.ai/docs/build-an-evaluation-dataset.md). For background, see [Datasets](https://docs.cloud.deepset.ai/docs/evaluation-datasets-and-metric-groups.md#datasets).

## Evaluate a Conversation from a Trace

If a trace shows an answer that looks wrong, you can run one of the pipeline's Experiments on the whole conversation that trace belongs to, without leaving the trace. The run judges every turn of the conversation and appears in the Experiment's run history. For steps and results, see [Evaluate a Conversation from a Trace](https://docs.cloud.deepset.ai/docs/trace-your-pipelines-ui.md#evaluate-a-conversation-from-a-trace). If a row errored because the LLM judge didn't conclude, see [Fix Sessions Where the Judge Didn't Conclude](https://docs.cloud.deepset.ai/docs/troubleshoot-evaluation.md#fix-sessions-where-the-judge-didnt-conclude).

## Related Information

-   [Share a Pipeline Prototype](https://docs.cloud.deepset.ai/docs/share-a-pipeline-prototype.md)
-   [Collect User Feedback](https://docs.cloud.deepset.ai/docs/collect-feedback.md)
-   [Improving Your Question Answering Pipeline](https://docs.cloud.deepset.ai/docs/improving-your-qa-pipeline.md)

-   [Pipeline Evaluation](https://docs.cloud.deepset.ai/docs/pipeline-evaluation.md)
-   [Write and Test an Evaluation Metric](https://docs.cloud.deepset.ai/docs/write-and-test-evaluation-metrics.md)
-   [Run an Experiment](https://docs.cloud.deepset.ai/docs/run-an-experiment.md)

-   [Start with Suggested Evaluation Goals](https://docs.cloud.deepset.ai/docs/propose-evaluation-goals.md)
-   [Define Evaluation Goals on a Pipeline](https://docs.cloud.deepset.ai/docs/define-evaluation-goals.md)
