Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Set Up an AI Operator for a ServiceBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Add an AI Operator to a running Service to measure what it costs and how well it answers, then ask it for cheaper configurations.

About This Task​

Before you start, make sure you have:

  • A running Service you want to optimize.
  • An Experiment for that Service's pipeline, with evaluators that judge answer quality. If the pipeline has no Experiment yet, the Add AI Operator modal tells you so and links to where you can create one.

Each Service can have only one AI Operator. If a Service already has a goal, it still appears in the Add AI Operator list, but it's disabled.

Add an AI Operator​

  1. Go to Operate > AI Operator, or open a Service and click Set up AI Operator next to the actions menu.
  2. Click Add AI Operator.
  3. Select the running Service you want to optimize.
  4. Select the Experiment whose evaluators judge this Service's answers.
  5. Under Goal, select what to Optimize for.
  6. Optional: For each score Metric, set a threshold on the scale that Metric declares, for example 4 on a 1–5 scale. Leave a field empty to hold that Metric to the baseline. To learn how thresholds are applied, see AI Operator.
  7. Click Save to create the goal.
  8. Click Save to create the goal.

If the goal can't be saved, check these causes:

  • An Evaluator of the Experiment doesn't declare its Metrics with @metrics. Add @metrics to the Evaluator.
  • The Experiment declares only label Metrics. Add an Evaluator with at least one score.
  • A threshold names a Metric that isn't a declared score, or sits outside that Metric's scale. Correct the threshold. Existing goals are checked the same way when you start a run. To learn how to declare Metrics, see Evaluation Metrics.

Result: The Service now has an AI Operator. Its row appears on the Operator list with a status based on its latest run, and the Service page shows an AI Operator button in place of Set up AI Operator.

Change the Metric Group of a Goal​

To score future runs with different Metrics, switch the goal to another Metric Group.

  1. Open the Operator page for the Service and click Edit goal.
  2. In the Metric Group for quality field, select another Metric Group.
  3. Click Update.

The new Metric Group applies from the next run. The Metric Group is locked while a run is open: the field is disabled, and the REST API answers a change with a 409 error. Stop the run or wait for it to end first.

Runs don't store the Metric Group they used, so past runs are read under the goal's current group. After a switch, older runs can show different metrics or fail to load their evaluation details. The new group must pass the same checks as when you add an Operator: every Evaluator declares its Metrics with @metrics, and at least one is a score.

View the Operator List​

To check which Services are being optimized, whether each has a baseline, and when each was last measured, go to Operate > AI Operator. The list shows every optimization goal in the workspace. For each row, you see:

  • Service: the Service the goal belongs to.
  • Goal: the goal written as a sentence, so you can read what it keeps and what it optimizes for.
  • Status: read from the latest measurement run (for example, whether a baseline exists or a run is in progress).
  • Last run: when the Operator last measured the Service.

Click a row to open that Operator's page.

The line under the Service name on the Operator page shows the pipeline name and the version of the live revision, for example my-pipeline · v295.

Measure a Baseline​

Measure a baseline to record the Service's current cost, latency, and quality. You need a baseline before you can request suggestions, and it gives you numbers to compare them against.

A baseline run needs at least 9 Sessions (conversations). The AI Operator splits them 70/30 and holds out 30% to judge each suggestion, so 9 Sessions leave 3 held out. If the Service has fewer Sessions than the minimum, the run is refused. The same minimum applies when you run on a Dataset: the Dataset needs at least 9 test cases. A run with fewer than 50 Sessions is marked as a small sample, and its verdicts are less reliable.

  1. Open the Operator page for the Service, either from the Operator list or the Service page.
  2. Choose the pipeline version you want to measure. Each time you open the run dialog, it starts on the live version, so the version you see is the version the run is baselined on.
  3. Choose the conversations to measure on: the newest conversations this version answered, or From a dataset. You need at least 7 complete conversations, or a dataset with at least 7 test cases from this pipeline. With a Dataset, every Session is measured. Sessions recorded on another version are replayed on this version first. If a single-turn Session already has an earlier replay on this version, for example from a Session Replay Suite, an Experiment, or an earlier run, the AI Operator reuses it instead of replaying it again. See AI Operator. For details on choosing conversations and following the run, see Start an Optimization Run.
  4. Start the measurement run. The Operator page switches to the new run so you can follow its progress. If the run is refused, see Fix a Refused Run.
  5. Review the baseline once the run finishes: cost, latency, and quality. The numbers appear in the What it costs today card. To see the conversations the run used, click Conversations in this run. If the run failed or shows a Small sample tag, see Read a Run.

Measuring a baseline scores every conversation with every evaluator of the Experiment. An organization runs at most six evaluations at the same time across Experiments, comparisons, and AI Operator runs, so a baseline over many conversations can take a while to finish. The run status stays in progress until all evaluations complete. For how waiting evaluations are queued and retried, see Run an Experiment.

If a Run Ends Before the Baseline Is Measured​

If a measurement run is stopped or fails before it records the baseline, the baseline card doesn't keep loading. It shows the message "The baseline was not measured: this run ended before it got that far." Start a new measurement run to get baseline numbers.

Inspect the Baseline Replay​

A baseline run replays recorded conversations against the pipeline version you chose. To check what the baseline answered for each conversation, open the replay inspector from the baseline card. It lists the replayed sessions and opens on On the baseline, which shows the traces of the baseline's answers. To compare the baseline's answers with the original ones, click As recorded to switch to the original session traces.

Each session in the list has a status tag:

  • Waiting: the run is still in progress and the baseline hasn't reached this session yet.
  • Replayed: the baseline replayed this session.
  • Could not replay: the session was excluded. Hover over the tag to see the reason.
  • Not replayed: the run ended before the baseline reached this session, so it won't be replayed. Hover over the tag, or open the session on the baseline side, to see the reason.

Follow a Run​

To check that a run measures the version you intended and to see where it stands, use the run progress card on the Operator page.

The card header names the run's baseline version, for example Baseline: my-pipeline · v295. If the version is not the one you meant to measure, you can tell as soon as the run starts.

The card lists the steps of the run. While the run is in progress, each step shows its current state. When the run ends, each step keeps what it did:

  • Sessions replayed, which you can still click to open.
  • Configurations proposed.
  • Configurations tested.
  • Total run time.

If a run fails or you stop it, the card keeps its steps, marks the step where the run ended, and shows the reason below the steps.

Inspect the Baseline Replay​

A run compares the Service's sessions against a baseline on the version you selected. Sessions that were recorded on a different version, or on no known version, are replayed on the baseline version first. To see which sessions were replayed, why, and what the baseline answered, open the replay inspector:

  1. On the Operator page, find the run progress card.
  2. In the first step, click the replay count. While the replay is running, it reads, for example, Replaying the baseline on sessions from other versions: 6 of 9. After the replay is done, it reads Replayed on N sessions from other versions.
  3. Select a session in the list to read its spans, including the input and output of each span.
  4. Use the toggle to choose which conversation you read:
    • As recorded: the session as it was originally answered.
    • On the baseline: the same turns as the baseline version answered them.

For each replayed session, the inspector shows:

  • Session ID: the identifier of the session.
  • Status: Waiting if the baseline has not reached the session yet, Replayed if the replay is done, or Could not replay if it failed. Hover over Could not replay to see the error.
  • Reason: why the session is replayed, for example Recorded on v293, baseline is v295, or that a turn records no version.

Result: You can verify which sessions were replayed on the baseline and compare each baseline answer with the recorded one.

Inspect the Sessions the Baseline Is Replayed On​

Before it measures anything, a measurement run replays the baseline on every test case that has at least one turn answered by another pipeline version or with no pipeline version recorded. A setup script on its own doesn't cause a replay. To check which sessions are replayed, or to compare what the baseline answered with what was recorded, open the replayed sessions from the run's progress.

  1. Open the Operator page for the Service and find the run in progress.
  2. In step 1 of the run progress, click the replay count (for example, Replaying the baseline on sessions from other versions: 3 of 11).
  3. In the Session inspector, select a session. To see what happened in the session, use the span tree and the span input and output view, which work the same way as for traces.
  4. Check the session's details: its ID, the version it was recorded on, and its status.
  5. Use the toggle to switch between As recorded and On the baseline. As recorded shows the turns of the original session. On the baseline shows the turns the baseline produced when it was replayed on that test case.

A session has one of these statuses:

  • Waiting: the baseline hasn't been replayed on this session yet.
  • Replayed: the baseline was replayed, and its turns are available under On the baseline.
  • Could not replay: the replay failed, so the session has no baseline turns.

Result: You can see which sessions the baseline was replayed on and compare its answers with the recorded ones side by side.

If you work with the API, the sessions endpoint of a run (GET .../runs/{run_id}/sessions) returns baseline_session_id and baseline_query_ids for each replayed test case. baseline_session_id is the session the baseline produced, and baseline_query_ids lists that session's turns in replay order. The list is empty until the baseline has been replayed on the test case. The endpoint also returns replay_reason for each session: OTHER_VERSION when a turn was answered by a pipeline version other than the baseline's, NO_VERSION when a turn has no pipeline version, or null when the test case is measured as recorded.

View Past Runs​

To compare what the Operator measured and suggested over time, or to look back at an earlier run, use the Runs table on the Operator page. The table lists every run of the Service's Operator, newest first. It appears once the Service has two or more runs.

For each run, the table shows:

  • Started: when the run started. Hover over the time to see the exact date. The newest run has a Latest tag, and the run currently shown on the page has a Viewing tag.
  • Version: the pipeline version the run measured. If the version isn't known, the table shows a placeholder instead.
  • Conversations: which conversations the run measured: sampled, hand-picked, or from a dataset.
  • Tested: how many configurations the run tested out of the maximum, for example "2 of 3 configurations".
  • Result: how the run ended. One of Running, Found an improvement, No improvement, Inconclusive, Stopped, or Failed.

The table shows 10 runs at first. To load 10 more, click Show older runs.

You can open an earlier run by clicking it in the Runs table.

Review, Accept, or Decline a Candidate​

To check a proposed configuration before you decide, use the Proposed candidates section on the Operator page.

  1. Click a candidate card to open its side panel.
  2. Review the Evaluation scores and Optimization run results tabs. To compare the candidate with the baseline configuration, click View diff in the summary at the top.
  3. Accept or decline the candidate. Both actions open the YAML diff against the baseline configuration.

Fix a Refused Run​

An AI Operator run can be refused when some of the Sessions it would use can't be replayed, or when too few Sessions remain. The error message names each affected Session by its name and its id, in the format '<name>' (<session id>). Several Sessions can share a name because a name is usually the Session's first query. Use the id to tell them apart.

Typical reasons for a refusal include:

  • A Session has turns that recorded no inputs because content tracing was off, so its requests can't be replayed.
  • A Session was recorded on a different pipeline from the one the Service runs.
  • Too many Sessions failed when the baseline was replayed on them.
  • Too few Sessions are left to hold out some for judging a candidate. A run needs at least 7 Sessions or test cases, counted after any that are left out. This can happen when recent traffic is used and Sessions are dropped because they recorded no inputs. For the run size limits, see Start an Optimization Run.

To fix a refused run:

  1. Read the error message and note the Session ids it lists.
  2. On the Operator page, check the run's Left out list. The Session column names each Session, and the Why column gives the reason it was left out.
  3. Remove the Session from the Dataset, or fix the cause (for example, turn on content tracing for the Service so new Sessions record their inputs).
  4. Start the run again. If the run is refused again, repeat these steps for the Sessions the new error message lists.

If you start runs through the API, a refused Dataset run returns a 422 response. Besides errors and test_case_ids, the response lists the affected Sessions in session_ids. Other refusals, such as too many Sessions that couldn't be replayed or too few Sessions left, name each left-out Session by name and id in the error message only. Use these ids to find the Sessions and their traces. To learn where to view traces, see Trace with Built-In Traces.

Fix a Replay That Was Not Recorded​

Use these steps when a baseline run or a candidate fails with a message that says how many replayed turns did not reach search history. The AI Operator replays conversations on the baseline and on each candidate configuration, then judges them. Before it judges a replay, it checks that every replayed turn is readable in search history. It checks repeatedly, waiting longer between checks, for about two minutes in total. This makes sure each conversation is judged whole, because judging only part of a conversation produces scores that can't be compared with the other side.

If some turns still aren't readable when the wait ends, the baseline run or the candidate fails with a message that says how many replayed turns did not reach search history. Nothing is judged from the partial conversation.

To recover:

  1. Check that the Service is running and that its pipeline records traces. Replayed turns can only be judged when they're written to search history.
  2. Start the baseline run again, or request suggestions again, to replay the conversations.
  3. If the failure repeats, open the traces for the Service to see whether recent turns appear. To learn where to view them, see Trace with Built-In Traces.

Read a Run​

To find out why a run failed or to check what a run measured, open the run view on the Operator page. It shows how far a measurement or optimization run has progressed and which conversations it used.

Find Out Why a Run Failed or Stopped​

When a run fails or is stopped, the run view shows the reason under the step the run ended on, not in a separate banner. For example, if the optimizer returns no candidates, the message Failed here. The optimizer returned no candidates. appears in red under the Propose configurations step.

A run that you stopped shows Stopped here before it finished. in amber under the step it was on when it stopped.

Open the Conversations a Run Used​

The run view lists the conversations the run measured in a Conversations in this run section. The section is collapsed by default. To see the conversations, click its header.

Even when the section is collapsed, its header shows:

  • The number of conversations.
  • The pipeline version the run measured.
  • How the conversations were chosen: sampled from the Service's traffic, or taken from a dataset. For a dataset run, the header gives the dataset's name. If the dataset was deleted or its name is still loading, the header says that the run took a dataset.

Check for a Small Sample​

If a run measured only a few conversations, its results are less reliable. In that case, the conversations header shows an amber Small sample tag. Hover over the tag to read the explanation.

Find Which Dataset a Run Used​

To compare runs, look at the Sessions column in the run history. For a run that measured a dataset, the column shows the dataset's name. For a run that sampled conversations, it shows how they were chosen.

Review Suggestions​

Once a baseline exists, you can ask the AI Operator for optimization suggestions:

  1. On the Operator page, request suggestions.
  2. Review each suggested configuration together with its validated cost and quality. Label evaluations, such as goal_reached, are listed under the quality gates but never gated. If a suggestion shows a Labels changed tag, check the replies before you accept it. See Review a Suggestion's Evaluation Scores. Quality doesn't rule out a suggestion for now, so check that each Metric is at or above its floor before you use it.
  3. To try a suggestion yourself, click Copy YAML in the suggestion's header to copy its configuration. The AI Operator does not apply or promote any suggestion automatically — you decide what to do with it.
  4. To decide whether the Service should use a suggestion, click Decline or Accept in the row of actions at the bottom right of its card. A card shows only the actions available for that suggestion. For details, see Act on a Suggestion.
  5. To get a new set of suggestions after a run, click Suggest again. This button appears only when the run can propose more suggestions. Review the new suggestions the same way, comparing each one's validated cost and quality.

Suggest Again​

If a run ends without a suggestion you can use, or the suggestions don't fit, ask the AI Operator for another round on the same run. A run's time and cost go mostly into picking and replaying conversations and measuring the baseline. Suggesting again reuses all of that: the run keeps its conversations and their split, its baseline replays, and its measured baseline. Only proposing and testing new candidates run again, under a fresh deadline.

Suggest again and Run again do different things:

  • Suggest again reopens the latest run and tests only new candidates.
  • Run again starts a new run from the beginning, including a new baseline measurement. Use it to measure the Service again, for example after you publish a new pipeline version.

The Suggest again button appears next to Run again only when the Service's latest run meets all of these conditions:

  • It has ended or failed. It can have ended with no improvement, an inconclusive result, a failure while proposing or testing, or a presented candidate.
  • It measured a baseline quality. If it didn't, use Run again.
  • None of its candidates is being promoted or was accepted. Once an accept starts, the Service no longer runs the baseline.
  1. On the Operator page, open the run and click Suggest again. A success notification appears and the run reopens. The run's status shows it as in progress until the new candidates are tested.
  2. Wait for the new round to finish. Each candidate shows the proposal round it came from.
  3. Review the candidates of the latest round together with the measured candidates from earlier rounds.

If you work with candidates through the API, each candidate has a proposal_round field: 1 for the run's first round, and one more for each further round.

Ask Again with the API​

To reopen a run programmatically, send a POST request to the run's propose-again endpoint:

curl --request POST \
  "https://<API_HOST>/<path-to-the-goal>/runs/<RUN_ID>/propose-again" \
  --header "Authorization: Bearer <YOUR_API_KEY>"

The response is the updated run. The request can fail with these status codes:

  • 404: the run doesn't exist.
  • 409: the run can't be reopened because it's still in progress, it isn't the goal's newest run, one of its candidates is being promoted or was accepted, or it never measured its baseline.

What to Expect​

  • The AI Operator tries something new. It is told which techniques earlier candidates used, how each one ended, and why you or the AI Operator declined it. It isn't shown their comparison scores, which come from the Sessions held out to judge candidates, so it can't tune toward them. A proposal that repeats the configuration of an earlier candidate is dropped before it is measured.
  • A presented candidate has to win again. If the run had presented a candidate, that candidate goes back to being measured and is ranked together with the new round's candidates.
  • An empty round doesn't lose your results. If a round produces no candidate that can be measured, the run ranks the measured candidates from earlier rounds instead of failing.

Review Label Changes in a Comparison​

Some evaluator metrics return a label, such as whether a goal was reached or an answer was grounded, instead of a numeric score. When the AI Operator compares a candidate with the baseline, it includes each label metric next to the quality gates. Use it to see whether the candidate changed how your Service's answers are judged, even when its scores pass.

For each label metric, the comparison shows:

  • Baseline: how many baseline rows carried each label value.
  • Candidate: how many candidate rows carried each label value.
  • Compared rows: how many replies were labelled on both sides. Replies are paired by conversation and turn.
  • Changed rows: how many of the compared replies have a different label on each side. This catches cases where two replies flip in opposite directions, which leaves the counts equal.

If any label moved, the candidate is marked as having changed labels, so you know to review it. This flag never rejects a candidate: nothing declares which label value is good, so only the numeric quality gates decide whether a candidate is eligible.

Baselines recorded before label rows were kept don't have paired rows. For these baselines, the compared and changed rows are empty, and the flag compares only the counts of each value. To get paired rows, measure a new baseline.

Ask for New Suggestions on an Ended Run​

If a run ended without an improvement, came back inconclusive, or failed, reopen it with Suggest again. See Suggest Again for the conditions and what the run reuses. Suggest again reopens the same run and asks the AI Operator for new suggestions. The run keeps the conversations it picked, the replays, and the baseline scores it already measured, so only the new suggestions are tested. This saves the time and cost of measuring everything again.

  1. Open the Operator page for the Service.
  2. In the latest run, click Suggest again next to Run again. Hover over the button to see what the run reuses.
  3. Wait while the Operator proposes and tests new suggestions. The page refreshes the run's progress on its own.

Result: The run reopens at the proposing step. When it finishes, the new suggestions appear ranked together with the earlier ones from the same run.

Suggest again appears only when the latest run has ended, didn't present a candidate, and has a measured baseline quality. If the baseline wasn't measured, or the run already presented a candidate, use Run again instead. Run again starts a new run and measures everything from scratch.

To act on a suggestion from the latest run, click Accept to roll out its configuration, or click Decline to dismiss it. You can accept or decline suggestions only for the latest run. Suggestions from earlier runs are read-only.

Get a Suggestion's Results Through the API​

To compare a suggested configuration with the run's baseline in your own tooling, request the suggestion's results. The response uses the same shape as the results of an Experiment, so you can read both the same way.

Send a GET request to this path on your Service's deployment:

/optimization-goals/{goal}/runs/{run}/candidates/{candidate}/results

The response contains:

  • metrics: the suggestion's value for each Metric and its delta from the baseline, in the order of the goal's Metric Group.
  • pipeline_metrics: duration, tokens, and cost, aggregated the same way as metrics.
  • test_cases: one entry for each validation Session the run held out. The baseline side is that Session, or the baseline's replay of it if the Session was recorded on another version. The suggestion side is the suggestion's replay of the Session.
  • status: the comparison's progress: REPLAYING, SCORING, ENDED, or FAILED.
  • pipeline_id, metric_group_id, baseline_experiment_run_id, and candidate_experiment_run_id: the identifiers you need to open the Sessions and the underlying scoring runs. The two run ids are null until the baseline is queued and the suggestion's replay has finished.

The results are partial while the baseline's or the suggestion's evaluation is still running. Poll the endpoint until status is ENDED or FAILED. The status is FAILED when either evaluation failed or is gone, or when the suggestion ended without being evaluated.

The endpoint returns a 404 response if the suggestion doesn't exist, belongs to a different run, or if the Service's pipeline no longer exists.

Score a Suggestion Again Through the API​

If a suggestion's evaluation failed, came back with failed results, or ran out of time, you can score it again without starting a new run. The AI Operator keeps the suggestion's replay and only judges again what didn't succeed.

Send a POST request to this path on your Service's deployment:

/optimization-goals/{goal}/runs/{run}/candidates/{candidate}/re-evaluate

The suggestion's replay must be complete, and the suggestion must have failed while it was being scored, finished scoring, been presented, or expired. If the run already ended, it reopens and ends again once the suggestion is scored, then ranks all its suggestions again.

The endpoint returns a 409 response with the code candidate_not_reevaluable in these cases: the suggestion is still being measured, was already accepted, declined, or removed, or its replay is incomplete. It also returns 409 if the run can't be reopened, for example because it isn't the goal's newest run or has an accepted suggestion. It returns 404 if the run or suggestion doesn't exist.

View a Suggestion's Evaluation Results​

To see exactly how a suggestion differs from the baseline, metric by metric, open its evaluation results. The results page lays out a suggestion against its baseline the same way an Experiment does, so you can judge whether the savings are worth any change in quality.

  1. On the Operator page, find the suggestion you want to check.
  2. Click View evaluation results on the suggestion. The Evaluation tab opens with Experiments highlighted.
  3. Read the results table. For each metric, including duration, tokens, and cost, it shows the suggestion's value, the change, and the baseline's value.
  4. To read the conversations behind the numbers, expand the test-case table. It lists each validation conversation with the Candidate and Baseline conversations side by side. Click a conversation to open it in the chat drawer.
  5. To inspect the raw scores, use the links to the suggestion's scoring run and the baseline's scoring run.

To check which technique the suggestion uses, its status, and how many validation conversations it was tested on, look at the page header.

To inspect the baseline's own scores, use the link on the baseline card on the Operator page. It opens the baseline's scoring run directly, because a baseline has nothing to compare against.

Result: While either side is still being scored, the page refreshes on its own. When scoring finishes, the table shows the final values.

Re-run a Suggestion's Evaluation​

If a suggestion's conversations were all replayed but its scoring failed, ran out of time, or left some scores failed, you can score it again without starting a new optimization run. The AI Operator reuses the replayed conversations and runs again only the scores that failed.

  1. On the Operator page, find the suggestion. The Re-run evaluation button appears on its card only when the suggestion can be scored again. Past runs don't offer it.
  2. Click Re-run evaluation.

Result: The card shows that the suggestion is being evaluated again. When scoring finishes, the new results replace the current ones. If the suggestion can no longer be re-evaluated, the panel says so and shows its current status.

Edit or Remove the Goal​

On the Operator page, you can:

  • Edit the goal to change what to Optimize for or the thresholds. You can't change the Experiment after you create the goal. Saving a goal that previously allowed some quality loss resets that allowance to zero, so proposed configurations must match the baseline's quality.
  • Remove the goal, which also removes the Service from the Operator list.

What To Do Next​