Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Start an Optimization RunBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

To find out whether a Service can cost less without answering worse, start an optimization run on its AI Operator page. A run measures what the Service costs today and how well it answers. It then proposes pipeline configurations and tests each one by replaying conversations against it.

About This Task​

A run measures conversations that one pipeline version answered. You choose which conversations it uses: the most recent ones, or the test cases of a dataset.

Nothing on your Service changes during a run. Each suggestion is tested on held-back conversations and is applied only if you accept it.

Prerequisites​

  • The Service serves a pipeline from the platform and has an AI Operator goal.
  • For a run on the most recent conversations, the chosen version has answered at least 7 complete conversations. The run holds some back to check suggestions against.
  • For a run on a dataset, the workspace has a dataset that includes this pipeline and has at least 7 test cases from this pipeline only.

Start a Run​

  1. Open the Service and go to its AI Operator page.
  2. Click Run now. If the Service already has runs, click Run again.
  3. Choose the pipeline version to measure. The version the Service runs today is selected by default.
  4. Set the maximum number of configurations the run may propose. Each configuration is replayed on real model spend, so this number limits the cost of the run.
  5. Under Conversations to measure, choose Most recent or From a dataset, and complete the options for your choice. For the options, see Measure the Most Recent Conversations and Choose a Dataset.
  6. Click Start run. The AI Operator page shows its progress.

Measure the Most Recent Conversations​

Choose Most recent to measure the newest conversations the chosen version answered.

Enter how many conversations to measure. The field is sized from what the version has answered:

  • It defaults to 50, or to the number the version has if that's fewer.
  • It can't go below the minimum of 7 or above the maximum the platform allows. The field shows the allowed range if you enter a number outside it.
  • If the version has answered fewer than 7 complete conversations, the modal says so and you can't start the run. Send the Service more queries, or choose another version, and try again.
  • If you enter fewer than 50, the modal warns that the results are marked low confidence. The run still shows suggestions, but treat a small sample's numbers as a hint.

Changing the version updates how many conversations are available.

Choose a Dataset​

Choose From a dataset to measure the test cases of a dataset instead of live traffic. Use this to measure the same conversations every time, so that runs are comparable.

The list shows the workspace's datasets that include this pipeline, each with its number of test cases. Some datasets are listed but can't be selected, and the list shows the reason:

  • The dataset has conversations from other pipelines.
  • The dataset has fewer than 7 test cases.

If you pick a dataset with fewer than 50 test cases, the modal warns that the results are marked low confidence.

If the platform can't measure some test cases, it refuses the run and lists every refused test case with the reason. The run is also refused if the test cases left out bring the dataset below 7. Fix or remove those test cases, or choose another dataset, and start the run again. For help, see Fix a Refused Run.

Use a Dataset Imported from an Archive​

To measure the same conversations in another workspace, use a dataset that was exported to a zip archive and imported there. An imported dataset works like any other dataset when you choose From a dataset. The archive carries the whole conversation of each test case, so the import works in a workspace that never recorded the original Sessions.

When you import an archive, you pick one pipeline, and the import replays every test case on that pipeline. Choose that pipeline's Service to run on the imported dataset. The platform stores the imported conversations under new Session ids, so they don't appear in search history or in your usage dashboards.

Follow a Run​

To check that the run is progressing and spot anything that affects how far you can trust its results, follow its steps on the AI Operator page.

  • Baseline: To see how far a dataset run has got, check the baseline step. It shows the progress of the baseline replay, for example "1 of 4". The platform replays the baseline on test cases that other versions answered, so that every test case is measured on the same version. When the replay is done, the step says the baseline is being measured.
  • Small sample banner: A warning banner appears on runs that measured fewer than 50 conversations. It's informational only, and the run continues. The suggestions of such a run are tagged Low confidence.

If scoring a baseline or candidate takes longer than expected, other evaluations may be using every available slot. An organization runs at most six evaluations at the same time, and this limit covers every evaluation in the organization, including experiments and the scoring of a run's baseline and candidates. The run waits until a slot is free.

  • Baseline: To see how far a dataset run has got, check the baseline step. It shows the progress of the baseline replay, for example "1 of 4". The platform replays the baseline on test cases that other versions answered, so that every test case is measured on the same version. If an earlier replay on the baseline version already covers a single-turn test case, the platform reuses it instead of replaying it, and it counts toward the progress. Earlier replays can come from a Session Replay Suite, an Experiment, or an earlier run. If the platform reuses a replay for every test case, the baseline moves straight to being measured. When the replay is done, the step says the baseline is being measured.
  • Failed or stopped runs: If a run fails or you stop it, the reason appears under the step where it ended, not in a banner. For example, a failed run shows "Failed here." followed by the cause, such as "The optimizer returned no candidates.", under Propose configurations. A stopped run shows "Stopped here before it finished." in amber under its step.
  • Small sample tag: A run that measured fewer than 50 conversations shows an amber Small sample tag on the header of the conversations panel. Hover over the tag to read the explanation. The tag is informational only, and the run continues. The suggestions of such a run are tagged Low confidence.
  • Created time and expiry: To see how long a run has been open and when it expires, read the line under the run's steps. It shows when the run was created, for example "Created 14 hours ago", next to an expiry notice. A run that lasts longer than 20 hours expires, along with anything not tested by then. Only the unfinished run expires; the AI Operator itself stays. Hover over the notice to see the exact date and time this run expires. If the run expires before it finishes, start a new run.

The steps of the run appear in the Runs card, on the right of the AI Operator page. Each step has the same width and its title sits under its icon, so the steps don't resize as their text changes.

Switch Between Runs​

To compare what an earlier run tested and suggested, pick it in the Runs card. The Current card on the left keeps showing the latest run, so you can compare the older run with today's cost.

  1. On the AI Operator page, open the run picker in the header of the Runs card.
  2. Select the run you want to see. The card shows the steps, suggestions, and conversations of that run.
  3. To return to the newest run, click Back to latest run next to the picker.

Read the Suggestions​

To decide which configuration to accept, read the cards under Proposed pipeline configurations in the Runs card.

  • The title of a folded card names the technique and the value it sets. Tags under the title show Low confidence, Not comparable, and the status of the suggestion.
  • The new cost per conversation and its percentage change from the baseline appear on the right of the card.
  • Unfold the card to read the recommendation and the rollouts. Use Copy YAML next to the configuration to copy it.
  • If the run ended early or proposed nothing, the section says so.

Collapse the Baseline and Conversations Sections​

To focus on the part of a run you're reading, collapse the What it costs today and Conversations in this run sections.

  • What it costs today is open when you load the page, because every suggestion is compared against these numbers.
  • Conversations in this run starts collapsed. Its header still shows how many conversations the run measured.

Click anywhere on a section header to expand or collapse it. The headers don't show an arrow.

Read the Conversations Panel​

To check which conversations the run measured and which ones it tested suggestions on, use the Conversations card. It splits the conversations into the half the optimizer was shown and the half held back to check suggestions.

In the Runs card, the conversations of a run appear after the suggestions as a summary line. Click View conversations to open the table in place.

To check which conversations the run measured and which ones it tested suggestions on, open the Conversations in this run panel. It splits the conversations into the half the optimizer was shown and the half held back to check suggestions.

The panel is collapsed by default, so the run's steps and suggestions stay in view. Click the header to expand it. The header stays visible when the panel is collapsed and shows:

  • The number of conversations.
  • The pipeline version the run measured.
  • How the conversations were chosen. For a dataset run, this is the name of the dataset. If the dataset was deleted or its name is still loading, the header says a dataset was used.
  • The Small sample tag, if the run measured fewer than 50 conversations.

In the expanded panel:

  • Dataset runs show each test case by name.
  • A Replayed tag marks a test case on which the baseline was replayed. This includes test cases where the run reused an earlier replay on the baseline version instead of replaying again.
  • The Left out tab lists each test case that the run excluded, with the reason.
  • While the baseline is still replaying, the panel says the conversations aren't split yet. It reloads as the replay progresses and shows the split when it's ready.
  • Click a conversation to open it in the trace inspector. Dataset runs read these traces by pipeline, because their conversations may not come from this Service.

Find a Dataset Run in the Run History​

To tell which runs measured which dataset, check the Sessions column of the run history. Runs on the most recent conversations show how the conversations were chosen. Runs on a dataset show the dataset's name.

Act on a Suggestion​

To decide whether a Service should use a suggestion, use the actions at the bottom right of its card:

  • Decline: Marks the suggestion as declined. The Service keeps its current configuration.
  • Accept: Applies the suggestion to the Service. If applying an accepted suggestion failed earlier, the button reads Retry accept.

A card shows only the actions that are available for that suggestion in its current state. Decline and Accept can be unavailable while another decision is in progress. Hover over them to see the reason.

Review a Suggestion's Evaluation Scores​

To decide whether a suggestion is safe to accept, expand its Evaluation scores panel. The panel compares the suggestion with what the Service does today on every evaluation metric the run measured. It has two parts: quality gates and label evaluations.

Quality Gates​

A quality gate is a numeric score with a floor that a suggestion has to clear. The gate table lists each score today, the score of the suggestion's configuration, and whether the suggestion passed. The panel header summarizes the result, for example "1 of 1 passed". A suggestion that misses a gate can't be presented.

Label Evaluations​

Some evaluations don't produce a score. They sort each reply into a label, for example resolved or unresolved for goal_reached. To learn more about labels, see Scores and Labels. No label is declared good or bad, so the platform never gates on them. It shows them so that you can check them yourself. Hover over the label tag next to a metric name to see why.

The label table appears under the gate table, with one row for each label metric:

  • Today: How many replies got each label on the baseline, for example 1 resolved · 1 unresolved.
  • This configuration: How many replies got each label on the suggestion, in the same format.
  • Changed replies: How many replies got a different label from the suggestion than from the baseline, out of the replies labeled on both sides, for example 1 of 2. The cell says None if no reply changed. It shows — if the baseline was measured before replies were paired, so the platform can't compare them. Check this column to find the replies to review before you accept the suggestion.

If the run found no numeric quality gate, the panel shows the "no quality floor" message and still lists the label evaluations under it.

Runs recorded before label evaluations were compared show the gate table only.

Check Replies When Labels Changed​

When at least one reply got a different label from the suggestion, a Labels changed tag appears in the panel header. It stays visible while the panel is collapsed, so you can see it when scanning a list of suggestions. The tag is a prompt to review the replies before you accept the suggestion, not a verdict. Hover over it for a reminder. It doesn't change the "passed" count or whether the suggestion can be presented.

  1. Expand the Evaluation scores panel and find the rows with a Changed replies value other than None.
  2. Click the link in the panel header to open the evaluation run, and compare the labels the baseline and the suggestion gave to the same replies.
  3. Decide whether the changes matter for your Service, and accept or decline the suggestion.

What To Do Next​