Compare Two Pipeline VersionsBeta
Find out whether a change to your pipeline, such as a new prompt or model, made it better or worse.
About This Task
To find out whether a new pipeline version beats your current one, start an Experiment that compares a Candidate against a Baseline on one Dataset, scored by one Metric Group. Unlike an experiment that only measures existing sessions, this kind of Experiment can replay conversations: it reuses the conversations a version already has for the Dataset's test cases and replays only the shortfall. When it finishes, you see whether the Candidate did better or worse than the Baseline, find the test cases behind the change, and fix evaluations that failed. For the concepts behind it, see Candidate and Baseline Experiments.
To measure sessions your pipeline already had instead, see Run an Experiment.
Prerequisites
- A Dataset with test cases. A Dataset is a collection of test cases that you keep for measuring your pipelines.
- A Metric Group in your workspace. A Metric Group is a set of metrics that scores the results together. If you don't have one, the New Experiment dialog tells you to create one in Metrics first.
- At least one pipeline version to use as the Candidate.
Start a New Experiment
- In the Evaluation section, click Experiments, and then click New Experiment.
- In Name, type a name that says what you compare, for example new prompt against production.
- In Dataset, choose the Dataset whose test cases both sides answer.
- In Metric Group, choose the Metric Group that scores the results.
- Under Candidate, choose the pipeline and the version you want to evaluate.
- Under Baseline, choose what to compare against:
- Pipeline version: Choose a pipeline and a version. It can be another version of the Candidate's pipeline. The Candidate and the Baseline must be different versions.
- Recorded conversations: Use each test case's own conversation, as the Dataset recorded it. Nothing is replayed for the Baseline.
- In Repetitions per test case, enter how many conversations each side scores for every test case. Conversations that already ran on that version are reused, and only the shortfall is replayed.
- In Replay, choose which requests of each test case's conversation to send: First message only (the default) or Whole conversation. Use First message only when a single question tests your change. Use Whole conversation when the answer depends on follow-up messages.
- Check the preview below the form. For each side, it shows how many conversations the side reuses and how many it replays. If every test case already has enough conversations on both sides, the preview says there is nothing to replay.
- Click Start Experiment.
The Experiment appears in the Experiments list with the status Replaying, and then Scoring. When the status changes to Finished, scoring is complete.
The dialog blocks Start Experiment and explains why in these cases:
- The Candidate and the Baseline are the same version. Pick a different version for one of them.
- The Experiment would score more results than the limit allows. Lower the repetitions per test case, or pick a smaller Dataset or Metric Group.
- A side would replay more conversations than the limit allows. Lower the repetitions per test case, or pick a smaller Dataset.
If the Experiment can't start, the dialog shows The Experiment did not start with the reason. Fix the issue and try again.
Review the Experiments List
Use the Experiments list to track a running Experiment and open its results when it finishes. In the Evaluation section, click Experiments to see every Experiment as a row. Each row shows:
- Experiment: The name you gave the Experiment.
- Candidate vs Baseline: The two sides of the comparison. For a pipeline version, the label shows the version's description next to its name. A Baseline that uses the Dataset's recorded conversations appears as Recorded conversations.
- Dataset: The test cases the Experiment uses.
- Metric Group: The metrics that score the results.
- Status: Replaying while conversations are replayed, Scoring while the Metric Group is applied, Finished when scoring is complete, or Failed when the Experiment didn't complete.
- A marker for failed evaluations: Some evaluations didn't complete. Hover over the marker to read "Some evaluations failed. Open the Experiment to rerun them." The marker is separate from the Failed status, which means the whole Experiment didn't complete.
- Created: When you started the Experiment.
Click a row to open the Experiment's detail page. If you have no Experiments yet, the page shows No Experiments yet and a prompt to start one.
Follow the Progress of Each Side
The detail page shows one card for the Candidate and one for the Baseline. Each card shows:
- The pipeline and version of the side, and a status: Replaying, Scoring, Finished, or Failed.
- How many conversations the side reuses and how many it replays.
- A progress bar for the replays, with the number of replays that finished and failed.
- An error message if the side failed.
The page refreshes on its own while a side is replaying or scoring. You can leave the page; the Experiment keeps running.
Read the Results Table
When a side is scored, the Results section shows one row for each Metric of the Metric Group. For each row, you see:
- The Candidate's value, in large type.
- The change against the Baseline, with an arrow for its direction and the difference in the Metric's own unit and as a percentage.
- The Baseline's value.
- The number of test cases scored on both sides, next to how many test cases the Dataset has. Hover over the info icon in the column header to learn how this count works. The change is based on these test cases.
The change is neutral on purpose: an arrow shows whether the value went up or down, but it's never green or red. A Metric doesn't say whether a higher value is better, so you decide whether the change is an improvement.
Below the Metrics of the Metric Group, the table shows the built-in measurements that every scoring run has: duration, tokens, and cost. Each one has an info icon that explains what it measures, and each is compared the same way as a Metric.
A label Metric has no average. For these Metrics, each side shows how many test cases got each label and their share.
If some evaluations of a Metric failed, the number of failed evaluations appears under the value.
The results also show how many results each side's Metric declared not applicable, for each Metric and each test case. A test case where a Metric only came back not applicable still appears, with that count, so you can tell it apart from an error or a test case that hasn't been scored yet.
Metrics Grouped by Evaluator
The results table has an Evaluator column. Its cell spans all the Metrics of one Evaluator, so the Evaluator's name appears once and each Metric shows by its key alone. To see the full name in the form Evaluator · key, hover over the Metric.
- Duration, tokens, and cost share one Built-in cell.
- An Evaluator whose only key is
valueis a legacy single-metric Evaluator. It's named in the Evaluator column alone, with no separate Metric name.
Label Metrics
A label Metric shows for each side as label counts, for example verdict pass ×2. The expanded row of a test case shows the labels of both the Candidate and the Baseline, and its one-line summary shows the Candidate's labels.
To see how the labels moved from the Baseline to the Candidate, read the change in the results table, in the expanded row, and in the one-line summary. Each label shows its change in count and in its share of the side's labels, for example pass +3 (+43%). Use it to spot a shift, such as more fail labels, that a single count hides.
- The largest shifts come first.
- Labels that didn't change are left out.
- If no label changed, the change reads No change.
- While a side has no labels yet, the change shows
—.
Check the Values for Each Test Case
To find out why a value moved, expand the test cases section. It lists the test cases with the values of each Metric for the Candidate and the Baseline. Expand a test case to see its details and open the conversations that produced the values.
For a test case from an imported Dataset, viewing the Baseline's recorded conversation opens the imported conversation drawer. There you can read each turn and click Go to trace to open it in the Session inspector. Conversations that the Experiment replayed open from search history as usual. For details, see View an Imported Conversation and Its Traces.
Every test case lists every Metric of the Metric Group, so you can compare test cases line by line. If a Metric has no value for a test case, it shows —. A Metric has no value when its evaluation errored, when it did not apply to the test case, or when it's still being scored. When the Candidate has no value, the change arrow is hidden.
A test case with no values at all reads Not scored yet. The collapsed line of a test case shows Metric keys only. Expand the test case to see the Evaluator column and both sides' values.
Fix Failed Evaluations
When some evaluations fail, the side's card shows a warning with the number of failed evaluations. The results leave them out. If the warning says that evaluations failed in the Metric's code, fix the Metric first, because rerunning doesn't help until you do.
Use the actions on the page:
- Rerun failed evaluations: Scores the failed evaluations again. Evaluations that succeeded aren't repeated.
- Rescore: Scores both sides again with the Metric Group as it is now. Every evaluation runs and is billed again. Conversations aren't replayed. Use it after you change a Metric.
- Rename: Click the pencil icon next to the name to rename the Experiment.
- Change on a side's card: Pick a different pipeline version for the Candidate or the Baseline. The new version is scored on the same test cases. It reuses the conversations it already has and replays the rest. If a replay is needed, you confirm with Replay and change, and the side's current results are replaced.
- Delete: Deletes the Experiment and its results. The conversations and replays that it reused or started are kept.
Open a Side's Scoring Grid
To see exactly what was scored, click Open scoring grid on a side's card. If the Baseline uses recorded conversations from several pipelines, the card has one link for each pipeline.
The scoring run page has one row for each Metric. It lists the replays that produced the conversations of the side. For each replay, you can always open its setup and cleanup logs, which helps you find out why a replay failed or behaved unexpectedly.
Related Information
Was this page helpful?