Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

AI Operator Run Time and CostBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Most of an AI Operator run's time and token spend goes into replaying conversations. Two choices keep both down: the baseline you pick, and whether you propose again or start a new run.

Choose a Baseline That Avoids Replays​

When you measure a baseline on test cases from a Dataset, the AI Operator uses each test case's recorded session if that recording can stand for the baseline. If it can't, the AI Operator replays the test case on the baseline. A replay is a full run of your pipeline for every turn, so it takes time and uses tokens.

To avoid replays, choose as the baseline the pipeline version that answered your recorded test cases. If every turn of a session was answered by that version, the AI Operator measures the session as recorded.

The AI Operator replays the whole session in these cases. It never reuses part of a session, because a later turn depends on the earlier ones.

  • OTHER_VERSION: at least one turn was answered by a different pipeline version than the baseline's.
  • NO_VERSION: at least one turn has no pipeline version, so nothing says which configuration answered it.

A setup script on a test case doesn't force a replay. When the AI Operator tests candidate configurations, it still runs the setup script before each replay.

Replays Are Reused​

You don't pay twice for the same replay. If an earlier replay of the test case on the baseline version exists, from a Session Replay Suite, an Experiment, or an earlier run, the AI Operator uses it instead of replaying again. It reuses a replay only if all of these are true:

  • The test case has a single turn.
  • The replay is complete and still in search history.
  • The baseline version answered it.
  • It ran the test case's current setup script.

Otherwise, the AI Operator replays the test case again. It replays multi-turn test cases itself so that the baseline and the suggestions it tests are replayed the same way. The replays it runs on single-turn test cases are available to Experiments and later runs. For details, see Replays.

To see what a replay produced, see Measure an AI Operator Baseline.

If a Replay Doesn't Appear​

The AI Operator judges a conversation only when every replayed turn is readable in search history. This way, you never compare a partial conversation with a complete one. If turns don't appear within about two minutes, the baseline run or the candidate fails instead of being judged. To recover, see Fix a Replay That Was Not Recorded.

Propose Again Instead of Starting a New Run​

If a run ends without a usable suggestion, or its suggestions don't fit your goal, you don't have to start over. Ask the run to propose again. Each attempt at proposing is a round.

A new round keeps the run's conversations, baseline replays, and baseline evaluation, so it skips the slowest work and goes straight to proposing. It's faster and cheaper than a new run. Choose Run again only when you want a new baseline, for example because the Service changed.

A run can propose again when all of these are true:

  • The run has ended.
  • It's the newest run for its goal.
  • It measured a baseline.
  • None of its candidates is being promoted or was accepted.

To ask for another round, see Ask an AI Operator for New Suggestions.

What to Expect from a New Round​

  • Earlier suggestions stay in the race. Candidates from earlier rounds are ranked with the new ones when the run concludes, so a better earlier configuration isn't lost. A candidate the run already presented has to win again.
  • The AI Operator learns from what you declined. It tells the optimizer which techniques earlier rounds tried and how each one ended, including why a candidate was declined. It doesn't share their scores, so the optimizer can't tune toward the conversations it's judged on.
  • Repeats are skipped. A proposal that repeats an earlier configuration is dropped before it's measured.
  • A round without results doesn't fail the run. If no new candidate can run, the run ranks the measured candidates of earlier rounds instead.