Evaluation ExperimentsBeta
An experiment is how you answer the same quality question repeatedly: you run a set of metrics over your pipeline's sessions, change your pipeline, and run it again. To compare two pipeline versions fairly, you run them on the same test cases.
How an Experiment Run Works
An experiment is a named set of metrics you run over your pipeline's sessions. For example, an experiment called "Groundedness and cost" could combine an LLM judge for groundedness with a code metric for token usage. To keep related metrics together and reuse them, group them in a Metric Group. For details, see Evaluation Datasets and Metric Groups.
Like metrics, experiments belong to your workspace, not to a single pipeline. You define the set of metrics once and use it to judge any pipeline in the workspace. Each run records which pipeline it judged.
Each time you run an experiment, you pick the sessions to measure. The run gives you a scorecard for each metric, with its distribution and any sessions worth a closer look, such as where the metric broke, a gap between sources, or a rare answer. From a scorecard, you can inspect the sessions behind it and see each turn and its rationale.
Comparing Runs
Two runs are comparable only if they measure similar sessions. If one run covers simple questions and the other covers hard ones, the difference in scores reflects the questions, not your pipeline. To compare two pipeline versions fairly, both need to answer the same or similar queries. The simplest way to guarantee that is a Candidate and Baseline Experiment.
Candidate and Baseline Experiments
When you want to know whether a new pipeline version beats the current one, compare them on exactly the same questions instead of comparing two separate runs. Create an Experiment that sets a Candidate against a Baseline. Using the same test cases on both sides means any difference in scores comes from the pipeline, not from the questions.
An Experiment of this kind combines four things:
- Dataset: The test cases both sides answer. A test case is one recorded conversation, together with optional setup and cleanup scripts that run around every replay of it.
- Metric Group: The set of metrics that scores them. Both sides use the same one, so the results are comparable.
- Candidate: The pipeline version you want to evaluate.
- Baseline: What you compare the Candidate against: another version of the same pipeline, or the Dataset's recorded conversations.
You also choose repetitions per test case: how many conversations each side scores for each test case. Several repetitions help you see how much a pipeline's answers vary from one run to the next.
Replays
A replay sends a test case's requests to a pipeline version again to produce a new conversation. You choose whether a replay sends only the first message or the whole conversation. A recorded conversation is the conversation a test case already holds. Conversations that already exist on a pipeline version are reused. The platform replays only the shortfall, which saves time and cost. Before you start, the New Experiment dialog previews how many conversations each side reuses and how many it replays. It blocks the start when the Experiment would exceed a limit. When the Baseline is the Dataset's recorded conversations, nothing is replayed for the Baseline.
To tell whether your results are ready, check the Experiment's status: Replaying while conversations are replayed, Scoring while the Metric Group is applied, and Finished or Failed.
Results
The Experiment is finished when both sides are scored. The results show the Candidate's value for each metric, the change against the Baseline, and the number of test cases scored on both sides. You can also ask the assistant for an AI Summary of how the Candidate compares to the Baseline. For steps, see Get an AI Summary of an Experiment.
To help you tell "didn't apply" apart from "errored" or "not scored yet", the results show, for each side, metric, and test case, how many results a metric declared not applicable. A test case where a metric returned only not-applicable results still appears in the results with that count.
A label metric has no mean, so its change against the Baseline is how each label's count and share of the side's labels moved, for example pass +3 (+43%). Results group metrics by the Evaluator that produced them, and every test case lists every metric of the Metric Group. A metric with no value for a test case shows —. For how to read these results, see Read the Results Table.
To start an Experiment, see Compare Two Pipeline Versions.
Related Information
Was this page helpful?