Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

AI OperatorBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

An AI Operator is an optimization goal set on one Service. Use it when you want to lower what a Service costs without lowering the quality of its answers. It measures what the Service costs and how well it answers, then suggests cheaper configurations for you to review.

What an AI Operator Does​

Each Service can have at most one AI Operator. When you add one, you choose:

  • The running Service you want to optimize.
  • The Experiment whose evaluators judge the quality of that Service's answers.
  • The quality the Service must keep while being optimized.

The AI Operator then measures a baseline for the Service: its cost, latency, and quality on the pipeline version you choose, based on a set of conversations. Once it has a baseline, you can ask it for optimization suggestions. The AI Operator proposes alternative, usually cheaper, pipeline configurations and validates that they still meet the quality you set. For now, quality doesn't rule out a suggestion: each Metric still shows the suggestion's score against the quality floor, but a suggestion below the floor can still be ranked and presented. Check each Metric before you use a suggestion.

Suggestions are advisory only. The AI Operator does not apply or promote configurations by itself — you review what it finds and decide what to do with it.

Switching the Metric Group​

You can switch the Metric Group that judges an AI Operator's goal. The new group applies from the next run. Runs don't store the Metric Group they used, so past runs aren't re-scored: they are read under the goal's current group, and older runs can show different metrics or fail to load their evaluation details. The Metric Group is locked while a run is open, because the run would otherwise score its candidates with Metrics its baseline wasn't scored on. To switch it, see Set Up an AI Operator for a Service.

Where You Find It​

The AI Operator has two entry points:

  • Operator list (go to Operate > AI Operator): to check on all optimization goals in the workspace, use this list. It shows each goal's Service, the goal written as a sentence, a status read from the latest measurement run, and when it last ran. Click a row to open that Operator.
  • Service page: to set up or open the AI Operator for a specific Service, use the Service page. A Service with no goal shows a Set up AI Operator button next to the actions menu. A Service that already has a goal shows an AI Operator button instead, which opens its Operator page directly.

AI Operator Suggestions​

To see what a Service costs now and what each optimization run found, open its AI Operator page. The page starts with the Service Cost History card, whose header shows the live cost per conversation over the past week. Below it, the Current and Runs cards show the baseline and the selected run, and the Proposed candidates section lists the configurations the run suggests. To read the cost card, see View Cost and Quality Trends on the AI Operator Page.

Current​

The Current card shows what the live pipeline costs now. You can compare an older run against today's numbers.

  • Measured version and cost per conversation: the top row shows the pipeline version the latest run measured, for example pipeline · v3, and its cost per conversation. Hover over the cost to see the total cost and the number of conversations measured.
  • Latest measurement: the measured numbers of the latest run, followed by Evaluation of the sessions. A folded breakdown shows where the money goes by component, with the models each component called, its calls, tokens in and out, cost, and share.

Runs​

Use this card to check what one run tested and suggested. You can switch runs using the run picker in the card header. When you view an older run, a Back to latest run link appears next to the picker.

Suggestion Cards​

Each proposed candidate is a compact card under Proposed candidates. The recommended candidate comes first, with a blue outline. Earlier rounds use the same cards in their folded section. A card shows:

  • Title: the technique and the value it sets, for example <technique> to <value>, with the value highlighted. If the Operator doesn't report a target value, the title shows the technique name only.
  • Tags: a second line shows Low confidence when the run measured a small sample, Not comparable when the suggestion can't be compared with the baseline, and the status of the suggestion. The Tested status isn't shown because it's the normal state of a finished suggestion.
  • New cost per conversation: the cost per conversation the suggestion reached when tested on held-back conversations, with its percentage change from the baseline.

The card also shows the composite quality score, the candidate's score compared with today's. The score is red when the candidate misses the composite threshold, yellow when it drops but still passes, and green when it improves.

Click a card to open a side panel with Evaluation scores and Optimization run results tabs. A summary at the top has a View diff button.

Key Ideas​

  • One goal per Service: you can't add a second AI Operator to a Service that already has one. The existing Service still appears in the Add AI Operator list, but it's disabled.
  • Experiment-backed quality: the quality threshold the Operator must keep comes from the evaluators defined in an Experiment. A pipeline with no Experiment can't get an AI Operator until you create one.
  • No quality loss: a proposed configuration must not lose quality compared with the current configuration (the baseline). Each score Metric is gated on its own, either against a threshold you set or against the baseline. For details, see How Quality Gates Work.
  • Fair quality comparisons: some evaluators answer not applicable on conversations where their question doesn't apply. To compare like with like, the AI Operator averages each score metric of the baseline and a suggestion over only the conversations both of them scored, then checks the quality floor against those averages. A good suggestion isn't rejected because the two scored different conversations. Runs recorded before scores were kept per conversation are compared as before.
  • Baseline before suggestions: before the AI Operator can propose anything, it needs a baseline measurement run on a chosen pipeline version using a set of conversations, split between training and held-out sets. The optimizer reads the training half before it suggests changes. The held-out half is kept back to measure the baseline and every suggestion, so scores on it come from conversations the optimizer never saw.
  • Small runs are allowed: a run can start from as few as 9 conversations (Sessions), so you can start optimizing a Service before it has much traffic. The AI Operator splits them 70/30: the larger share is shown to the optimizer, and the smaller share is held out to judge each suggestion. With 9 conversations, each suggestion is judged on 3 held-out conversations, so its verdict is less reliable than on a larger run. Runs with fewer than 50 conversations are marked as a small sample.
  • Every run is kept: the Operator keeps each run and its suggestions. The latest run is the one the Operator is working on now. You can open an earlier run to see what it measured and suggested, but only the latest run's suggestions can be accepted or declined.
  • Whole conversations only: when the AI Operator replays conversations on a baseline or a candidate, it waits until every replayed turn is readable in search history before it judges them. This keeps a partial conversation from being scored and then compared with a complete one, so the baseline and candidate quality you compare are based on the same complete conversations. If turns don't appear within about two minutes, the baseline run or the candidate fails instead of being judged. See Fix a Replay That Was Not Recorded.
  • Labels are flagged, not gated: some evaluator metrics return a label, such as whether a goal was reached, instead of a score. Nothing declares which label value is good, so the AI Operator never uses labels to accept or reject a candidate. It compares them for you instead: for each label metric, you see how often each value came up on the baseline and on the candidate, and how many replies changed value. A candidate whose labels changed is flagged for your review but stays eligible.

How Quality Gates Work​

The AI Operator gates each suggestion on every score Metric of the Experiment's Evaluators, so one Metric can't collapse while the average holds. A threshold is set on the scale the Evaluator declares with @metrics, for example 4 on a 1–5 judge, so Metrics aren't limited to scores between 0 and 1. label Metrics are never gated.

A score Metric is gated like this:

  • With a threshold: the suggestion fails the gate when its score is below the threshold.
  • Without a threshold: the suggestion fails the gate when its score is more than 20% of the scale below the baseline. That's 0.2 on a 0–1 scale and 0.8 on a 1–5 scale. A suggestion that is below the baseline but within that 20% gets a warning and isn't blocked.

Each gate in the results has one of these statuses:

  • Failed: the suggestion didn't meet the gate and is blocked.
  • Warning: the suggestion is below the baseline but within the allowance. It isn't blocked, so check it before you use it.
  • Not measured: the Metric has no score on the side or sides the gate needs, for example because the Evaluator answered not applicable on every conversation. The Metric isn't gated. A Metric with a threshold is still gated when only the baseline has no score.

If the baseline has no score at all, or a suggestion has none, quality can't be measured and the suggestion fails the gate.

Goals saved before thresholds existed have none, so they hold every score Metric to the baseline until you set thresholds. To set thresholds, see Set Up an AI Operator for a Service.

How the Baseline Is Replayed on Test Cases​

When a measurement run uses test cases from a Dataset, the AI Operator measures the baseline on each test case's recorded session if that recording can stand for the baseline. Otherwise, it replays the baseline on the test case and measures the replay. A replay is a full run of your pipeline for every turn, so it takes time and uses tokens. To avoid replays, choose as the baseline the pipeline version that answered your recorded test cases. The AI Operator replays only when it has to. It also reuses earlier replays of a test case on the baseline version, including first-message replays that an Experiment ran, as long as the test case has a single turn. For details, see Replays.

The AI Operator replays the baseline on a test case for one of these reasons:

  • OTHER_VERSION: at least one turn of the recorded session was answered by a pipeline version other than the baseline's version. A later turn depends on the earlier ones, so the AI Operator replays the whole session and never reuses part of it.
  • NO_VERSION: at least one turn of the recorded session has no pipeline version. Nothing says which configuration answered it, so the AI Operator replays it.

If every turn was answered by the baseline's version, the AI Operator measures the test case as recorded and doesn't replay it.

A setup script on a test case doesn't force a replay. A session recorded on the baseline's version already ran in the state that the script recreates. When the AI Operator tests candidate configurations, it still runs the setup script before each replay.

See What a Replay Produced​

To compare a test case as it was recorded with the way the baseline answered it, and to see why the AI Operator replayed it, open the replayed sessions from the measurement run's progress or list them through the API. For steps and the fields the API returns, see Set Up an AI Operator for a Service.

  • baseline_session_id: the session the baseline produced when replayed on the test case. It's null until the replay happens.
  • baseline_query_ids: the turns of that session, in replay order. It's empty until the baseline has been replayed on the test case.
  • replay_reason: why the baseline is replayed, either OTHER_VERSION or NO_VERSION. It's null when the test case is measured as recorded, and for runs frozen before this field was kept.

Use these fields to compare a test case as it was recorded with the way the baseline answered it, and to see why the AI Operator replayed it.

  • Replays are reused: when a dataset test case was recorded on another version, the AI Operator has to replay the baseline version on it. If an earlier replay of that test case on the baseline version already exists, from a Session Replay Suite, an Experiment, or an earlier run, the AI Operator uses it instead of replaying again. This saves time and model spend. It reuses a replay only if the test case has a single turn, the replay is still in search history and complete, it was answered by the baseline version, and it ran the test case's current setup script. Otherwise, it replays the test case again. The replays the AI Operator runs itself on single-turn test cases are also available to Experiments and later runs, so they can reuse them instead of replaying those test cases again.
  • Why single-turn only: the AI Operator sends every turn as it was recorded, while Session Replay builds each follow-up question from its own replay's answers, so the two methods differ from the second turn on. To avoid mixing replay methods between the baseline and the suggestions it tests, the AI Operator replays multi-turn test cases itself.

Proposing in Rounds​

When a run ends without a usable suggestion, or its suggestions don't fit, you can have it propose again instead of starting a new run. Each attempt at proposing is a round. Most of a run's time and cost goes into picking and replaying conversations and measuring the baseline. A new round keeps the run's frozen conversations, baseline replays, and baseline evaluation, so it skips that work and goes straight to proposing. This makes a new round faster and cheaper than a new run. Run again starts over with a new baseline instead.

A run can propose again when all of these are true:

  • The run has ended.
  • It's the newest run for its goal.
  • It measured a baseline.
  • None of its candidates is being promoted or was accepted.

How rounds work:

  • Candidates from earlier rounds stay on the run and are ranked with the new ones when the run concludes. A candidate the run already presented has to win again.
  • The AI Operator tells the optimizer which techniques earlier rounds tried and how each one ended, including why a candidate was declined or whether you removed it. It doesn't share their validation scores, because that would let the optimizer tune toward the conversations it's judged on.
  • A proposal that repeats an earlier candidate's configuration is dropped before it's measured.
  • Each candidate records its proposal round, so you can tell which attempt produced it: 1 for the first proposal, and one more each time the run proposes again.
  • If a round yields no candidate that can run, the run ranks the measured candidates of earlier rounds instead of failing.

To ask for another round, see Suggest Again.

  • Gates and labels: the AI Operator gates a suggestion only on numeric scores, which must clear the quality floor. Label evaluations, such as resolved or unresolved, have no good or bad value, so they're shown for you to review but never block a suggestion. To read them, see Review a Suggestion's Evaluation Scores.

  • Suggestions read like Experiments: each suggestion has an evaluation results page that compares it against its baseline, metric by metric, in the same layout as an Experiment. Use it to decide whether a suggestion keeps enough quality to be worth adopting. To open it, see View a Suggestion's Evaluation Results.