Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Evaluation ExperimentsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

An experiment is how you answer the same quality question repeatedly: you run a set of metrics over your pipeline's sessions, change your pipeline, and run it again. To compare two pipeline versions fairly, you run them on the same test cases.

How an Experiment Run Works​

An experiment is a named set of metrics you run over your pipeline's sessions. For example, an experiment called "Groundedness and cost" could combine an LLM judge for groundedness with a code metric for token usage. To keep related metrics together and reuse them, group them in a Metric Group. For details, see Evaluation Datasets and Metric Groups.

Like metrics, experiments belong to your workspace, not to a single pipeline. You define the set of metrics once and use it to judge any pipeline in the workspace. Each run records which pipeline it judged.

Each time you run an experiment, you pick the sessions to measure. When the run ends, the platform computes one aggregate for each metric, weighted by session, and stores it with the run. The scoring-run page shows that aggregate as a scorecard for each metric, so everything that reads the run shows the same numbers. A scorecard shows:

  • The headline: the mean for a score metric, or the most common label and its share for a label metric.
  • How many of the judged sessions the metric valued.
  • How many sessions broke and how many were not applicable (N/A).

While the run is still scoring, the page shows no scorecards. They appear when the run ends. From a scorecard, you can open the breakdown to see the distribution, the sessions worth a closer look, and each turn and its rationale.

Comparing Runs​

Two runs are comparable only if they measure similar sessions. If one run covers simple questions and the other covers hard ones, the difference in scores reflects the questions, not your pipeline. To compare two pipeline versions fairly, both need to answer the same or similar queries. The simplest way to guarantee that is a Candidate and Baseline Experiment.

Run Aggregate​

When a run ends, the platform stores its Aggregate: what the run found, with one entry per metric. Every place that shows a run's numbers reads the same Aggregate, so they agree. The scoring-run read endpoint of the REST API returns it. The Aggregate is empty (null) while a run is still going, and for a run whose Aggregate wasn't computed.

The Aggregate counts Sessions, not rows. A three-turn Session weighs the same as a one-turn Session. Each metric's entry holds:

  • Value: For a score metric, the mean of the Sessions' means, in the metric's declared units. A 1-to-5 judge reads 1 to 5, and nothing is normalized or clipped. For a label metric, the share of each label, averaged across Sessions.
  • Coverage: The share of judged Sessions that have a value.
  • Session counts: How many Sessions are valued, not applicable, unusable, errored, failed, or pending. Pending Sessions haven't reported yet and aren't counted as judged.
  • Evaluator versions: The versions of the Evaluator whose code ran for the metric.
  • Direction and range: The better direction and the declared range of a score.

The Aggregate holds values, not a verdict. Whether a value is good depends on a threshold you set elsewhere.

To read an Aggregate correctly:

  • A metric that is missing from the Aggregate is unknown. It doesn't mean the metric passed.
  • Failed, errored, unusable, and pending Sessions are counted separately. Check coverage before you trust a mean.
  • If the Evaluator versions that contributed disagree on better or range, that field is null.
  • If a metric's results disagree on kind, for example a score in some rows and a label in others, the metric has no value, and its shape is mixed.

Candidate and Baseline Experiments​

When you want to know whether a new pipeline version beats the current one, compare them on exactly the same questions instead of comparing two separate runs. Create an Experiment that sets a Candidate against a Baseline. Using the same test cases on both sides means any difference in scores comes from the pipeline, not from the questions.

An Experiment of this kind combines four things:

  • Dataset: The test cases both sides answer. A test case is one recorded conversation, together with optional setup and cleanup scripts that run around every replay of it.
  • Metric Group: The set of metrics that scores them. Both sides use the same one, so the results are comparable.
  • Candidate: The pipeline version you want to evaluate.
  • Baseline: What you compare the Candidate against: another version of the same pipeline, or the Dataset's recorded conversations.

You also choose repetitions per test case: how many conversations each side scores for each test case. Several repetitions help you see how much a pipeline's answers vary from one run to the next.

Replays​

A replay sends a test case's requests to a pipeline version again to produce a new conversation. You can choose if a replay sends only the first message, the whole conversation, or pauses to ask for approval. For details, see Replay With Automatic Approval.

A recorded conversation is the conversation a test case already holds. Conversations that already exist on a pipeline version are reused. The platform replays only the shortfall, which saves time and cost.

Before you start, the New Experiment dialog previews how many conversations each side reuses and how many it replays. It blocks the start when the Experiment would exceed a limit. When the Baseline is the Dataset's recorded conversations, nothing is replayed for the Baseline.

Replay With Automatic Approval​

Some pipelines pause and ask for approval before they continue, for example before an agent runs a tool or takes an action. A replay that sends only the first message stops at that question, so the conversation you score is shorter than the real one. To test the conversation up to and including the approval, use the First message, then auto-approve replay mode (FIRST_USER_MESSAGE_AUTO_APPROVE in the API).

In this mode, the platform replays the first request of the test case. After each reply, a replay guide decides what happens next. The replay guide is an LLM pipeline that plays the user:

  • If the pipeline asks for something to be approved, the guide writes an approval. The platform sends it as the next request in the same replay conversation.
  • If there is nothing to approve, or the pipeline only suggests a follow-up that isn't a direct continuation of the original request, the replay ends.

The guide sees the replay so far. It also sees the messages the real user sent later in the original conversation, as a hint of what that user approved. A replay sends at most five approvals by default. If the replay guide fails, the replay fails, so you never score a shorter conversation without knowing it.

Set Up the Replay Guide​

The replay guide runs inside your deepset instance, on the model credentials of a dedicated workspace. The conversation content stays on the instance, and your own model provider isn't billed for it. Until an administrator configures the guide, requests for the auto-approve replay mode return a 422 error. This applies to a single replay, a Replay Suite, and a comparison.

To configure the guide for an instance:

  1. Create a dedicated workspace for the guide.
  2. In that workspace, create a pipeline named Session-Replay-Guide (or another name that matches the setting below). Upload its YAML, mark a version as the default, and add the model provider secrets the pipeline needs.
  3. Set the instance settings in the following table.
SettingDefaultDescription
SESSION_REPLAY_GUIDE_WORKSPACE_IDNoneID of the workspace that hosts the guide pipeline. The auto-approve mode is unavailable until you set it.
SESSION_REPLAY_GUIDE_PIPELINE_NAMESession-Replay-GuideName of the guide pipeline in that workspace.
SESSION_REPLAY_GUIDE_TIMEOUT_SECONDS60How long the platform waits for the guide to answer.
SESSION_REPLAY_AUTO_APPROVE_MAX_FOLLOW_UPS5Maximum number of approvals sent in one replay.

The pipeline's inputs mapping can declare any of these inputs:

  • conversation: The replay so far, as a list of messages with role and content.
  • source_user_messages: The messages the original user sent after the first message.
  • query: An instruction to decide whether to approve.
  • messages: Sent empty. An Agent builds its user message from its own prompt.

The pipeline must return a JSON object somewhere in its output. To approve, return {"decision": "APPROVE", "message": "<the approval to send>"}. To end the replay, return {"decision": "END", "reason": "<why>"}.

To tell whether your results are ready, check the Experiment's status: Replaying while conversations are replayed, Scoring while the Metric Group is applied, and Finished or Failed.

Results​

The Experiment is finished when both sides are scored. The results show the Candidate's value for each metric, the change against the Baseline, and the number of test cases scored on both sides. You can also ask the assistant for an AI Summary of how the Candidate compares to the Baseline. For steps, see Get an AI Summary of an Experiment.

To help you tell "didn't apply" apart from "errored" or "not scored yet", the results show, for each side, metric, and test case, how many results a metric declared not applicable. A test case where a metric returned only not-applicable results still appears in the results with that count.

A label metric has no mean, so its change against the Baseline is how each label's count and share of the side's labels moved, for example pass +3 (+43%). Results group metrics by the Evaluator that produced them, and every test case lists every metric of the Metric Group. A metric with no value for a test case shows —. For how to read these results, see Read the Results Table.

The platform reads each metric's change against its declared direction, so a change counts as improved, regressed, unchanged, or unknown. See Metric Direction.

The Experiment's results in the REST API also carry a headline for each side: the normalized mean of each Metric and the Metric Group's Composite, so you can compare the Candidate and the Baseline with one number. The platform computes them when you read the results, with the Metric Group's current weights. For how they are calculated, see Normalized Means and the Composite.

A side has no Composite in these cases:

  • The side spans several scoring runs, for example a Recorded conversations Baseline that covers several pipelines. There is no rule yet for merging runs into one Composite.
  • The Metric Group was deleted.
  • A weighted Metric has no normalized mean, as for any other run.

In each case, the headline says why.

To start an Experiment, see Compare Two Pipeline Versions.

Label Changes Against the Baseline​

The results chart how each label's share of the test cases moved against the Baseline: one bar per label, growing right in blue when the label's share grew and left in orange when it shrank. A label growing isn't good or bad on its own: more grounded answers is an improvement, more not grounded answers isn't. For how to read the chart, see Label Metrics.

Built-In Pipeline Metrics​

Besides the metrics in your Metric Group, the results table and the test-case table include built-in rows that show what the pipeline spent to produce each session. To see whether a Candidate that scores better also costs more or runs slower, read these rows next to your quality metrics:

  • Duration: the pipeline's run time in seconds, summed over a session's turns.
  • Tokens: the tokens the pipeline's LLM calls used, shown as input and output together on one line, for example 1K in / 200 out. In is the tokens sent to the LLMs, such as the prompt and retrieved documents. Out is the tokens the LLMs generated. The Δ vs Baseline column shows the change for each part, for example +200 in (+25.0%) / +100 out (+100.0%). Comparing the two parts shows whether a cost change comes from a longer prompt or a longer answer.
  • Cost: what the LLM calls cost in USD. A session counts only if every call in it is priced, so unpriced sessions don't understate the spend.

Each value is the average over the sessions of a test case, and the change against the Baseline counts only test cases that have a run on both sides. To find which test cases make the Candidate cost more or run slower, read these values for each test case. For how to read the rows, see Read the Results Table.