Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Start an Optimization RunBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

To find out whether a Service can cost less without answering worse, start an optimization run on its AI Operator page. A run measures what the Service costs today and how well it answers. It then proposes pipeline configurations and tests each one by replaying conversations against it.

About This Task​

A run measures conversations that one pipeline version answered. You choose which conversations it uses: the most recent ones, or the test cases of a dataset.

Nothing on your Service changes during a run. Each suggestion is tested on held-back conversations and is applied only if you accept it.

Prerequisites​

  • The Service serves a pipeline from the platform and has an AI Operator goal.
  • For a run on the most recent conversations, the chosen version has answered at least 16 complete conversations. The run holds some back to check suggestions against.
  • For a run on a dataset, the workspace has a dataset that includes this pipeline and has at least 16 test cases from this pipeline only.

Start a Run​

  1. Open the Service and go to its AI Operator page.
  2. Click Run now. If the Service already has runs, click Run again.
  3. Choose the pipeline version to measure. The version the Service runs today is selected by default.
  4. Set the maximum number of configurations the run may propose. Each configuration is replayed on real model spend, so this number limits the cost of the run.
  5. Under Conversations to measure, choose Most recent or From a dataset, and complete the options for your choice. For the options, see Measure the Most Recent Conversations and Choose a Dataset.
  6. Click Start run. The AI Operator page shows its progress.

Measure the Most Recent Conversations​

Choose Most recent to measure the newest conversations the chosen version answered.

Enter how many conversations to measure. The field is sized from what the version has answered:

  • It defaults to 50, or to the number the version has if that's fewer.
  • It can't go below the minimum of 16 or above the maximum the platform allows. The field shows the allowed range if you enter a number outside it.
  • If the version has answered fewer than 16 complete conversations, the modal says so and you can't start the run. Send the Service more queries, or choose another version, and try again.
  • If you enter fewer than 50, the modal warns that the results are marked low confidence. A small sample gives numbers you should treat as a hint.

Changing the version updates how many conversations are available.

Choose a Dataset​

Choose From a dataset to measure the test cases of a dataset instead of live traffic. Use this to measure the same conversations every time, so that runs are comparable.

The list shows the workspace's datasets that include this pipeline, each with its number of test cases. Some datasets are listed but can't be selected, and the list shows the reason:

  • The dataset has conversations from other pipelines.
  • The dataset has fewer than 16 test cases.

If you pick a dataset with fewer than 50 test cases, the modal warns that the results are marked low confidence.

If the platform can't measure some test cases, it refuses the run and lists every refused test case with the reason. Fix or remove those test cases, or choose another dataset, and start the run again.

Follow a Run​

To check that the run is progressing and spot anything that affects how far you can trust its results, follow its steps on the AI Operator page.

  • Baseline: To see how far a dataset run has got, check the baseline step. It shows the progress of the baseline replay, for example "1 of 4". The platform replays the baseline on test cases that other versions answered, so that every test case is measured on the same version. When the replay is done, the step says the baseline is being measured.
  • Small sample banner: A warning banner appears on runs that measured fewer than 50 conversations. It's informational only, and the run continues. The suggestions of such a run are tagged Low confidence.

Read the Conversations Card​

To check which conversations the run measured and which ones it tested suggestions on, use the Conversations card. It splits the conversations into the half the optimizer was shown and the half held back to check suggestions.

  • Dataset runs show each test case by name.
  • A Replayed tag marks a test case on which the baseline was replayed.
  • The Left out tab lists each test case that the run excluded, with the reason.
  • While the baseline is still replaying, the card says the conversations aren't split yet. It reloads as the replay progresses and shows the split when it's ready.
  • Click a conversation to open it in the trace inspector. Dataset runs read these traces by pipeline, because their conversations may not come from this Service.

What To Do Next​