Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

AI Operator Quality GatesBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

A proposed configuration must not lose quality compared with the current configuration, the baseline. The AI Operator checks this with quality gates.

Gates on Score Metrics​

The AI Operator gates each suggestion on every score Metric of the Experiment's Evaluators, so one Metric can't collapse while the average holds. A threshold is set on the scale the Evaluator declares with @metrics, for example 4 on a 1–5 judge, so Metrics aren't limited to scores between 0 and 1. label Metrics are never gated.

A score Metric is gated like this:

  • With a threshold: the suggestion fails the gate when its score is below the threshold.
  • Without a threshold: the suggestion fails the gate when its score is more than 20% of the scale below the baseline. That's 0.2 on a 0–1 scale and 0.8 on a 1–5 scale. A suggestion that is below the baseline but within that 20% gets a warning and isn't blocked.

Each gate in the results has one of these statuses:

  • Failed: the suggestion didn't meet the gate and is blocked.
  • Warning: the suggestion is below the baseline but within the allowance. It isn't blocked, so check it before you use it.
  • Not measured: the Metric has no score on the side or sides the gate needs, for example because the Evaluator answered not applicable on every conversation. The Metric isn't gated. A Metric with a threshold is still gated when only the baseline has no score.

If the baseline has no score at all, or a suggestion has none, quality can't be measured and the suggestion fails the gate.

Goals saved before thresholds existed have none, so they hold every score Metric to the baseline until you set thresholds. To set thresholds, see Set Up an AI Operator for a Service.

Fair Quality Comparisons​

Some evaluators answer not applicable on conversations where their question doesn't apply. To compare like with like, the AI Operator averages each score metric of the baseline and a suggestion over only the conversations both of them scored, then checks the quality floor against those averages. A good suggestion isn't rejected because the two scored different conversations. Runs recorded before scores were kept per conversation are compared as before.

Labels Are Flagged, Not Gated​

Some evaluator metrics return a label, such as whether a goal was reached or resolved or unresolved, instead of a score. Nothing declares which label value is good, so the AI Operator never uses labels to accept or reject a candidate. It compares them for you instead: for each label metric, you see how often each value came up on the baseline and on the candidate, and how many replies changed value. A candidate whose labels changed is flagged for your review but stays eligible. To read the comparison, see Review AI Operator Suggestions.

Suggestions Read Like Experiments​

Each suggestion has an evaluation results page that compares it against its baseline, metric by metric, in the same layout as an Experiment. Use it to decide whether a suggestion keeps enough quality to be worth adopting. To open it, see View a Suggestion's Evaluation Results.