Evaluation GoalsBeta
An evaluation goal states what good enough looks like for a pipeline: which metric to read, what a single session must reach to pass, and how many sessions must pass.
A metric gives you a value for every session. A goal turns that value into a pass or fail rule, so you can tell whether the pipeline meets your bar. A goal reads one metric from one Evaluator in your workspace. You define goals per pipeline, so two pipelines can hold different goals on the same Evaluator. Use goals to check whether a pipeline meets your quality bar before you rely on it, and to see which pipelines fall short.
The Pass Ruleβ
The pass rule decides whether a single session passes. It follows how the metric is declared:
- Score metric: A session passes when the mean of its values is at or above the goal's threshold if the score declares that higher is better, or at or below it if the score declares that lower is better.
- Label metric: A session passes when every one of its rows has one of the goal's passing values.
The goal reads the direction from the metric. If a later version of the metric flips it, the goal flips with it.
Minimum Pass Rateβ
The minimum pass rate is the share of sessions that must pass for the goal to be met, from 0 to 1. The default is 1, which means every session must pass. The goal is met when the number of passing sessions divided by the number of sessions with a value is at least the minimum pass rate. Sessions where the metric was not applicable, ended in error, or hasn't finished don't count either way.
Managing Goalsβ
You can find all the evaluation goals on a pipeline's Evaluation tab. Check the stage of the run and the elapsed time:
- Queued: The run is waiting to start.
- Preparing N Sessions: The platform is getting the Sessions ready for judging. The button doesn't fill yet.
- Judging done/total: The platform is judging. A judgment is one Session judged by one Evaluator, so the total is the number of Sessions times the number of Evaluators. Judgments reused from earlier runs count as done right away, so the bar can jump at the start.
Hover over the button to see how many judgments have errored, are running, and are waiting for a free slot. The list puts Missed goals first, then Met goals, then the rest, so what needs attention is at the top.
If the pipeline has no evaluation goals yet, define the first goal with the New Evaluation goal button. For the steps, see Define Evaluation Goals on a Pipeline.
Experiments and Datasets of the Pipelineβ
To review past Experiments or manage the Datasets that test your pipeline, open the Resources menu in the header and choose Experiments or Datasets. Each list opens inside the Evaluation tab, and β Evaluation goals above the list takes you back to the goals. Opening an Experiment or a Dataset from the list shows its details in the tab as well.
Both lists show only what belongs to this pipeline:
- Experiments: An Experiment is listed when either its Baseline or its Candidate ran on this pipeline.
- Datasets: A Dataset is listed when it has a test case recorded on this pipeline. The pickers you use when you create an Experiment, add to a Dataset, or copy test cases still offer every Dataset in the workspace.
Both lists use the same layout. The top bar holds the title, quick filters, and the actions for creating or bringing in items. The footer holds the actions for the selected rows and the page controls. Only the rows scroll, so the bar and the footer stay in view. A sort orders all pages, not only the page you see.
The Datasets list shows each Dataset's name, its number of test cases, when it was created, and when it was last updated. To delete Datasets, select their rows and click Delete selected in the footer. Experiments that already ran a deleted Dataset keep their results. To clear the selection, click Clear selection.
For how Experiments work, see Evaluation Experiments. For Datasets, see Evaluation Datasets and Metric Groups.
Sources of an Evaluation Runβ
Click Evaluate now to open a dialog that asks first what to run:
- Evaluate Evaluation goals: Judge every goal on the Sessions you pick.
- Run an Experiment: Run every goal on two versions of the pipeline over the same Sessions and compare them.
In both modes, you then choose where the Sessions come from. This lets you check a draft under the same rules as live traffic, or on a fixed set of Sessions:
- A time window: Today, Last 7 days, Last 30 days, or a Custom range. The dialog shows the Session count of the window, and you can sample the window to fewer Sessions. An untouched Last 7 days is the default run.
- A Dataset: Pick an existing Dataset, or build a new one. To build a new one, search the pipeline's Sessions, select the ones you want, and give the Dataset a name. The Dataset keeps those Sessions, so you can evaluate the same ones again later. The search covers the newest 200 Sessions of the pipeline.
If the Dataset has test cases with no Session recorded for this pipeline, for example test cases from another pipeline, the run replays them on the draft, and the dialog says so. The run then needs a draft.
The Session count, sampling, and replay options appear only when the platform can preview runs.
Size a Run Before You Start Itβ
A run is limited in how many judgments it can make, so it helps to know the size before you start. A preview counts, for a source, without starting anything or locking the pipeline:
- How many Sessions a time window holds, counted up to the window ceiling of 500, or how many usable test cases a Dataset has.
- How many of them the run would judge, after any sampling and the per-run limits.
- How many judgments that makes, which is the Sessions judged times the number of distinct Evaluators across the goals, and the limit that applies.
- For a Dataset, whether some test cases have no Session recorded on this pipeline, so they are replayed on the draft.
- For an Experiment, how many replays each side that runs a version would start.
The preview and the run use the same limits, so the count you see is the count the run judges. A preview doesn't refuse an over-limit run. The run itself does, and a window or a new Dataset is trimmed to stay within the limit.
To judge fewer Sessions of a time window, set a maximum number of Sessions. The run judges the newest ones up to that number, still within the per-run limits. A window run with a maximum counts as a custom run, even over the default 7 days, so it doesn't feed the default verdicts or the trend.
Experimentsβ
An Experiment compares two sides:
- Baseline: The version you compare from. It starts as the draft, which is the pipeline's draft version. If the pipeline has no draft, pick the Baseline yourself.
- Candidate: The version you compare with the Baseline. You pick it from the pipeline's versions.
In the dialog, each side has a label you edit in place and a version select. Labels tell the sides apart in the cards and in the run select, for example "Shorter prompt". Labels are at most 64 characters. The two sides can't use the same version.
An Experiment also sets how it replays the Sessions. The replay mode decides what each replay sends. It defaults to the first user message of each Session. Replays per test case sets how many Sessions are scored for each test case and side, from 1, the default, up to the maximum the platform allows. Replaying each test case more than once shows how much a result varies. Because every extra replay adds judgments and replays, a time window keeps that many times fewer test cases, so each side stays within the replay and scoring limits.
Choose Which Run You Seeβ
The run select in the header decides which run, or which Experiment, the cards' status and Pass or Fail come from. By default it shows the latest run on the last 7 days. Pick a past window run, a Dataset run, or an Experiment to see the goals judged on it. Switching the run keeps the page in place and loads only the statuses and counts.
When you pick an Experiment, each card shows both sides, for example "Draft Fail Β· Shorter prompt Pass". A side that is still scoring is marked as partial. On an Experiment, cards sort by the Candidate's verdicts, and the trend keeps showing the default 7-day series. The run select appears only when the platform can list past runs.
Statusesβ
When you evaluate a pipeline's goals, each goal gets one of these statuses. By default, the evaluation uses the sessions from the last 7 days:
- Met: The share of passing sessions reaches the minimum pass rate.
- Missed: The share is below the minimum pass rate.
- Not enough traffic: Fewer than 10 sessions had a value the goal could judge, so the goal can be neither met nor missed. Send more traffic to the pipeline and evaluate again.
- Not evaluated yet: The goal hasn't been evaluated.
- Evaluatingβ¦: An evaluation is running. A goal that already has a result keeps it until the run finishes. To follow the run, watch the progress on the Evaluate now button.
- Couldn't evaluate: The last evaluation failed. Click Evaluate now to try again.
A goal can also carry a Stale tag. It means the last evaluation can't answer the goal as it is now, for example after you changed its rule or Evaluator. Click Evaluate now to update it.
Trend Across Runsβ
To see whether a pipeline is getting better or worse against a goal, unfold the goal's card. Beside the passing rule, the card shows how the goal's value changed over its past runs:
- One run: The card shows the result as a number with its date, for example "92% of Sessions passed on 1 Nov."
- Two or more runs: The card shows a line chart with one point per run. A dashed line marks the goal's target, which is the minimum pass rate. Each point is marked Met or Missed, so you can see which runs reached the target. Hover over a point to see its date and value.
Percentages are rounded down, so a run that falls short of the target never appears on or above the target line.
A run needs at least 10 Sessions with a value to count. Runs with fewer appear as gaps in the line instead of as a point, and a note under the chart says how many were left out. If only one run has enough Sessions, the card shows it as a number, not a chart. If no run has enough Sessions, no trend appears.
A past run opens its scoring matrix in a page that slides in, so you can see how each session was scored.
AI Explanation of Missed Goalsβ
To find out why a goal missed without reading every failing session yourself, use the explanation the AI Assistant writes for each Missed goal. When the page shows a run's results, it asks the AI Assistant about each Missed goal. The explanation appears on the page itself, not in the Ask AI panel, and it doesn't add to the panel's conversation.
The assistant reads the goal's rule, the pass counts, and up to 50 failing Sessions from the run, each with its value and rationale. The rationale is cut to 400 characters. At most three goals are explained at the same time.
- In the goal list: Each Missed goal shows a one-line headline. While the assistant works, the row shows a loading state. If the explanation fails, the row shows a Retry button.
- On the Live tab: The full analysis appears under the verdict, marked Written by AI. It contains:
- Clusters of failing Sessions that share a pattern, with the number of Sessions in each. There are up to three clusters.
- A likely cause. Treat it as a hypothesis, not a confirmed diagnosis.
- Up to three example Sessions. Each example links to the Session's trace on the pipeline's Traces tab, where you can inspect what happened. The link opens the trace for single-turn Sessions only.
The page keeps each explanation for the run, goal, and rule it was written for, so reopening the page in the same browser session doesn't ask the assistant again. If you change a goal's rule or evaluate again, the page requests a new explanation.
If an Explanation Failsβ
If a row shows that the miss couldn't be explained, click Retry. The explanation fails when the assistant is unavailable or when its answer doesn't match the expected format, for example when it names a Session that didn't fail. If it keeps failing, the cause is on the assistant's side, not in your goal. The goal's status and pass rate aren't affected.
Related Informationβ
Was this page helpful?