Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Measure an AI Operator BaselineBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Measure a baseline to record the Service's current cost, latency, and quality. You need a baseline before you can request suggestions, and it gives you numbers to compare them against.

About This Task​

A baseline run needs at least 9 Sessions (conversations). The AI Operator splits them 70/30 and holds out 30% to judge each suggestion, so 9 Sessions leave 3 held out. If the Service has fewer Sessions than the minimum, the run is refused. The same minimum applies when you run on a Dataset: the Dataset needs at least 9 test cases. A run with fewer than 50 Sessions is marked as a small sample, and its verdicts are less reliable. To learn how the baseline works, see AI Operator Run Time and Cost.

Measure a Baseline​

  1. Open the Operator page for the Service, either from the Operator list or the Service page.
  2. Choose the pipeline version you want to measure. Each time you open the run dialog, it starts on the live version, so the version you see is the version the run is baselined on.
  3. Choose the conversations to measure on: the newest conversations this version answered, or From a dataset. You need at least 7 complete conversations, or a dataset with at least 7 test cases from this pipeline. With a Dataset, every Session is measured. Sessions recorded on another version are replayed on this version first. If a single-turn Session already has an earlier replay on this version, for example from a Session Replay Suite, an Experiment, or an earlier run, the AI Operator reuses it instead of replaying it again. For details on choosing conversations, see Start an Optimization Run.
  4. Start the measurement run. The Operator page switches to the new run so you can follow its progress. If the run is refused, see Fix a Refused or Failed AI Operator Run.
  5. Review the baseline once the run finishes: cost, latency, and quality. The numbers appear in the What it costs today card. To see the conversations the run used, click Conversations in this run.

Measuring a baseline scores every conversation with every evaluator of the Experiment. An organization runs at most six evaluations at the same time across Experiments, comparisons, and AI Operator runs, so a baseline over many conversations can take a while to finish. The run status stays in progress until all evaluations complete. For how waiting evaluations are queued and retried, see Run an Experiment.

If a measurement run is stopped or fails before it records the baseline, the baseline card shows the message "The baseline was not measured: this run ended before it got that far." Start a new measurement run to get baseline numbers. If the run failed or shows a Small sample tag, see Read an AI Operator Run.

Inspect the Baseline Replay​

A run compares the Service's sessions against a baseline on the version you selected. Before it measures anything, it replays the baseline on every test case that has at least one turn answered by another pipeline version or with no pipeline version recorded. A setup script on its own doesn't cause a replay. To see which sessions were replayed, why, and what the baseline answered, open the replay inspector.

  1. On the Operator page, find the run progress card, or open the replay inspector from the baseline card.
  2. In the first step of the run progress card, click the replay count. While the replay is running, it reads, for example, Replaying the baseline on sessions from other versions: 6 of 9. After the replay is done, it reads Replayed on N sessions from other versions.
  3. Select a session in the list to read its spans, including the input and output of each span. The span tree and the input and output view work the same way as for traces.
  4. Use the toggle to choose which conversation you read. The inspector opens on On the baseline:
    • As recorded: the session as it was originally answered.
    • On the baseline: the same turns as the baseline version answered them.

For each replayed session, the inspector shows:

  • Session ID: the identifier of the session.
  • Reason: why the session is replayed, for example Recorded on v293, baseline is v295, or that a turn records no version.
  • Status: a tag with one of these values:
    • Waiting: the run is still in progress and the baseline hasn't reached this session yet.
    • Replayed: the baseline replayed this session.
    • Could not replay: the replay failed or the session was excluded. Hover over the tag to see the reason.
    • Not replayed: the run ended before the baseline reached this session. Hover over the tag, or open the session on the baseline side, to see the reason.

Result: You can verify which sessions were replayed on the baseline and compare each baseline answer with the recorded one.

Read the Replay Through the API​

The sessions endpoint of a run (GET .../runs/{run_id}/sessions) returns these fields for each replayed test case:

  • baseline_session_id: the session the baseline produced when replayed on the test case. It's null until the replay happens.
  • baseline_query_ids: the turns of that session, in replay order. The list is empty until the baseline has been replayed on the test case.
  • replay_reason: OTHER_VERSION when a turn was answered by a pipeline version other than the baseline's, NO_VERSION when a turn has no pipeline version, or null when the test case is measured as recorded. It's also null for runs frozen before this field was kept.