Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Review and Re-Run Optimization EvaluationsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Open the baseline's evaluation to see how it scored on training and held-out conversations, and check how each suggestion's quality compares with the baseline.

About This Task​

An optimization run splits its conversations into two halves. The optimizer is shown the training half. The held-out half is kept back to measure the baseline and each suggestion. When you read an evaluation, knowing which half a conversation belongs to tells you whether a score comes from conversations the optimizer saw or from conversations it never saw.

The run scores the baseline and each suggestion (candidate) with the evaluators of the Service's Experiment. Some evaluators answer not applicable on conversations where their question doesn't apply, for example a verification metric on turns where the pipeline didn't verify anything.

To learn what an AI Operator is and how runs work, see AI Operator.

How Quality Is Compared​

The platform compares a suggestion with the baseline on one composite quality score. The composite is the mean of the score Metrics that both sides scored, each normalized to a range of 0 to 1. Label Metrics never count. Because the composite covers only what both sides scored, a good suggestion isn't rejected because the baseline and the suggestion scored different conversations. A baseline with some failed evaluations is judged on the evaluations that succeeded. To learn more, see the fair quality comparisons in AI Operator.

How Quality Gates Work​

Each suggestion is gated on its composite quality, against the baseline's composite and the allowed quality drop you set on the AI Operator goal. A suggestion fails when its composite falls below the baseline's composite reduced by that percentage. For example, with an allowed drop of 20%, a baseline composite of 0.80 can fall to 0.64. To learn how the gates work, see AI Operator Quality Gates. To set the allowed drop, see Set Up an AI Operator for a Service.

Check the composite quality and gate status of each suggestion

:

  • Failed: the composite fell below the floor and the suggestion is blocked.
  • Warning: an individual Metric scores below the baseline while the composite stays above the floor. The suggestion isn't blocked, so check that Metric before you use the suggestion.
  • Not measured: the Metric has no score where the composite needs one, for example because the evaluator answered not applicable on every conversation.

If the baseline or a suggestion has no score at all, quality can't be measured and the suggestion fails the gate.

Open the Baseline Evaluation by Split​

To see how the baseline scored on each conversation, and which half each conversation belongs to, open the baseline's evaluation.

  1. Open the Service and go to its AI Operator page.
  2. In the baseline card (What it costs today), find the quality section.
  3. Click View full evaluation.

A conversation is named after its test case, or after its first query if it has no test case. If the run left a test case out, it has no half and shows its name alone. When the baseline was replayed on a test case, the replay carries the same label as the original conversation.