Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Re-Run an Evaluation in an Optimization RunBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

To get fresh quality scores for a suggestion or for the baseline of an optimization run, run its evaluation again through the API. Use this when a metric came back not applicable or failed, or when you want every conversation judged again.

About This Task​

An optimization run scores the baseline and each suggestion (candidate) with the evaluators of the Service's Experiment. Some evaluators answer not applicable on conversations where their question doesn't apply, for example a verification metric on turns where the pipeline didn't verify anything. A re-run asks the evaluators again, so a result that was not applicable or failed is judged anew instead of being reused.

A re-run uses the evaluator versions the run was pinned to when it started, so the new scores stay comparable with the earlier ones.

How Quality Is Compared​

The platform compares each score metric of a suggestion and the baseline over the conversations both sides scored. For details, see AI Operator.

  • If an evaluator was not applicable on some conversations for one side, the platform leaves those conversations out of both averages. Without this, the baseline could be averaged over 2 conversations and the suggestion over 4.
  • The quality floor for each metric is checked against these paired averages.
  • A run recorded before the platform kept scores per conversation is compared as before.
  • The platform no longer rejects a suggestion because the two sides scored different sets of conversations.

Prerequisites​

  • The run has ended and is the newest run of the AI Operator goal.
  • No suggestion of the run has been accepted.
  • For the baseline: the baseline was evaluated.

Re-Run a Suggestion's Evaluation​

Send a POST request to .../optimization-goals/{goal}/runs/{run}/candidates/{candidate}/re-evaluate.

The platform judges every conversation of the suggestion again. If the evaluation has to wait for a free slot, the platform still judges it again when the slot is available.

Result: When scoring finishes, the new results replace the suggestion's earlier ones.

Re-Run the Baseline's Evaluation​

Send a POST request to .../optimization-goals/{goal}/runs/{run}/re-evaluate-baseline.

The baseline's configuration, conversations, and replays don't change. Only its scoring is repeated.

  1. The platform judges every conversation of the baseline again, on both the shown and the held-back half. Test cases the run left out stay out.
  2. The baseline's quality returns to pending and is filled in the next time the run is read.
  3. Suggestions that were scored on a whole replay are compared again with the new baseline scores, and the run is reopened at the evaluating step.
    • A suggestion with a finished evaluation keeps it.
    • A suggestion whose evaluation failed while it was being scored, or that expired, is evaluated again.
  4. If the run has no suggestion to compare, for example because the baseline evaluation failed or timed out, the run stays as it ended. After the scores are in, you can ask the AI Operator to suggest again.

Result: The baseline has fresh scores, and every suggestion that can be compared is compared against them.

Errors​

StatusCodeMeaning
404n/aThe run doesn't exist.
409baseline_not_reevaluableThe run hasn't ended, isn't the newest run, has an accepted suggestion, or has no evaluated baseline.
409evaluations_busyAnother evaluation is being started for this run or its Experiment. Try again in a moment.
503n/aThe platform couldn't queue the evaluation. Try again later.