Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Review AI Operator SuggestionsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Check what the AI Operator suggests, compare each suggestion with the baseline, and decide whether the Service should use it.

About This Task​

Once a baseline exists, you can ask the AI Operator for optimization suggestions. They appear in the Proposed candidates section on the Operator page. The AI Operator doesn't apply or promote any suggestion automatically: you decide what to do with it.

Review, Accept, or Decline a Suggestion​

  1. On the Operator page, click a suggestion card to open its side panel.
  2. Review the Evaluation scores and Optimization run results tabs, together with the suggestion's validated cost and quality. To compare the suggestion with the baseline configuration, click View diff in the summary at the top. Quality doesn't rule out a suggestion for now, so check that each Metric is at or above its floor before you use it. For the label evaluations, see Read Label Changes.
  3. To try a suggestion yourself, click Copy YAML in the suggestion's header to copy its configuration.
  4. To decide, click Decline or Accept in the row of actions at the bottom right of its card. A card shows only the actions available for that suggestion. Both actions open the YAML diff against the baseline configuration. For details, see Act on a Suggestion.

To get a new set of suggestions, see Ask for New Suggestions.

Read a Suggestion Card​

Each card under Proposed candidates shows the technique and value it suggests, the new cost per conversation with its change from the baseline, and a composite quality score. The recommended suggestion comes first, with a blue outline. Use these cues to decide how much to trust a card:

  • Low confidence: the run measured a small sample, so the result is less reliable.
  • Not comparable: the suggestion can't be compared with the baseline, so don't judge it by its cost change alone.
  • Score color: red means the suggestion misses the composite threshold, yellow means the score drops but still passes, and green means it improves.

Read Label Changes​

Some evaluator metrics return a label, such as whether a goal was reached, instead of a numeric score. When the AI Operator compares a suggestion with the baseline, it lists each label metric next to the quality gates, never gated. If a suggestion shows a Labels changed tag, check the replies before you accept it. For each label metric, the comparison shows:

  • Baseline: how many baseline rows carried each label value.
  • Candidate: how many candidate rows carried each label value.
  • Compared rows: how many replies were labelled on both sides. Replies are paired by conversation and turn.
  • Changed rows: how many of the compared replies have a different label on each side. This catches cases where two replies flip in opposite directions, which leaves the counts equal.

Baselines recorded before label rows were kept don't have paired rows. For these baselines, the compared and changed rows are empty, and the flag compares only the counts of each value. To get paired rows, measure a new baseline. For the scores of a suggestion, see also Review a Suggestion's Evaluation Scores.

View a Suggestion's Evaluation Results​

To see how a suggestion differs from the baseline, metric by metric, open its evaluation results. The page lays out a suggestion against its baseline the same way an Experiment does, so you can judge whether the savings are worth any change in quality.

  1. On the Operator page, find the suggestion you want to check.
  2. Click View evaluation results on the suggestion. The Evaluation tab opens with Experiments highlighted. The page header shows the suggestion's technique, status, and how many validation conversations it was tested on.
  3. Read the results table. For each metric, including duration, tokens, and cost, it shows the suggestion's value, the change, and the baseline's value.
  4. To read the conversations behind the numbers, expand the test-case table. It lists each validation conversation with the Candidate and Baseline conversations side by side. Click a conversation to open it in the chat drawer.
  5. To inspect the raw scores, use the links to the suggestion's scoring run and the baseline's scoring run.

To inspect the baseline's own scores, use the link on the baseline card on the Operator page. It opens the baseline's scoring run directly, because a baseline has nothing to compare against.

Result: While either side is still being scored, the page refreshes on its own. When scoring finishes, the table shows the final values.

Re-run a Suggestion's Evaluation​

If a suggestion's conversations were all replayed but its scoring failed, ran out of time, or left some scores failed, you can score it again without starting a new optimization run. The AI Operator reuses the replayed conversations and runs again only the scores that failed.

  1. On the Operator page, find the suggestion. The Re-run evaluation button appears on its card only when the suggestion can be scored again. Past runs don't offer it.
  2. Click Re-run evaluation.

Result: The card shows that the suggestion is being evaluated again. When scoring finishes, the new results replace the current ones. If the suggestion can no longer be re-evaluated, the panel says so and shows its current status. To do the same through the API, see Work with AI Operator Suggestions Through the API.