Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Review and Re-Run Optimization EvaluationsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Re-run an evaluation when a metric came back not applicable or failed, or when you want every conversation judged again.

About This Task​

An optimization run splits its conversations into two halves. The optimizer is shown the training half. The held-out half is kept back to measure the baseline and each suggestion. When you read an evaluation, knowing which half a conversation belongs to tells you whether a score comes from conversations the optimizer saw or from conversations it never saw.

The run scores the baseline and each suggestion (candidate) with the evaluators of the Service's Experiment. Some evaluators answer not applicable on conversations where their question doesn't apply, for example a verification metric on turns where the pipeline didn't verify anything. A re-run asks the evaluators again, so a result that was not applicable or failed is judged anew instead of being reused.

A re-run doesn't replay any conversation, so it's cheaper than starting a new run. The baseline and the suggestions keep the answers they already produced. A re-run uses the evaluator versions the run was pinned to when it started, so the new scores stay comparable with the earlier ones.

To learn what an AI Operator is and how runs work, see AI Operator.

How Quality Is Compared​

The platform compares each score metric of a suggestion and the baseline only over the conversations both sides scored, and checks the quality floor against those averages. To learn why, see the fair quality comparisons in AI Operator.

What Happens During a Baseline Re-Run​

Whether you start it on the page or through the API, a baseline re-run works like this:

  1. The platform judges every conversation of the baseline again, on both the shown and the held-back half. Test cases the run left out stay out.
  2. The baseline's quality returns to pending and is filled in the next time the run is read.
  3. Suggestions that were scored on a whole replay are compared again with the new baseline scores, and the run is reopened at the evaluating step.
    • A suggestion with a finished evaluation keeps it.
    • A suggestion whose evaluation failed while it was being scored, or that expired, is evaluated again.
  4. If the run has no suggestion to compare, for example because the baseline evaluation failed or timed out, the run stays as it ended. After the scores are in, you can ask the AI Operator to suggest again.

Prerequisites​

To re-run an evaluation:

  • The run has ended and is the newest run of the AI Operator goal.
  • No suggestion of the run has been accepted.
  • For the baseline: the baseline was evaluated.

Open the Baseline Evaluation by Split​

To see how the baseline scored on each conversation, and which half each conversation belongs to, open the baseline's evaluation.

  1. Open the Service and go to its AI Operator page.
  2. In the baseline card (What it costs today), find the quality section.
  3. Click View full evaluation.

A conversation is named after its test case, or after its first query if it has no test case. If the run left a test case out, it has no half and shows its name alone. When the baseline was replayed on a test case, the replay carries the same label as the original conversation.

Re-Run the Evaluation of a Suggestion​

Re-run a suggestion's evaluation when its scoring failed or expired, or when you want every conversation judged again. The AI Operator doesn't replay the suggestion. It judges the replays it already has.

On the AI Operator Page​

Re-run evaluation is offered on a suggestion card when both of these are true:

  • Every conversation of the suggestion was replayed.
  • The suggestion was scored (tested or recommended), expired, or failed while scoring.

To re-run the evaluation:

  1. On the AI Operator page, open the run that contains the suggestion.
  2. On the suggestion's card, click Re-run evaluation.
  3. Wait for the scores to appear on the card.

A suggestion that failed before its replay was complete doesn't offer the button, because there is nothing to judge yet.

Re-Run the Evaluation of the Baseline​

Re-run the baseline's evaluation when you want the baseline scored again, for example after you change an evaluator in the Experiment. The baseline's configuration, conversations, and replays don't change. Only its scoring is repeated, and the run's suggestions are compared with the new baseline scores.

On the AI Operator Page​

Re-run evaluation is offered next to View full evaluation when all of these are true:

  • The run is the latest run of the AI Operator and it has ended.
  • The baseline was evaluated and its scores are in.
  • You haven't accepted any suggestion of the run. This applies whether or not the run has suggestions.

To re-run the evaluation:

  1. Open the Service and go to its AI Operator page.
  2. In the baseline card, click Re-run evaluation next to View full evaluation.
  3. Keep the page open. The page keeps checking the run until the new scores are in, even though the run has ended.