Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Fix a Refused or Failed AI Operator RunBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Fix the Sessions that cause an AI Operator run to be refused, or recover a run that failed because replayed turns weren't recorded.

Fix a Refused Run​

An AI Operator run can be refused when some of the Sessions it would use can't be replayed, or when too few Sessions remain. The error message names each affected Session by its name and its id, in the format '<name>' (<session id>). Several Sessions can share a name because a name is usually the Session's first query. Use the id to tell them apart.

Typical reasons for a refusal include:

  • A Session has turns that recorded no inputs because content tracing was off, so its requests can't be replayed.
  • A Session was recorded on a different pipeline from the one the Service runs.
  • Too many Sessions failed when the baseline was replayed on them.
  • Too few Sessions are left to hold out some for judging a candidate. A run needs at least 7 Sessions or test cases, counted after any that are left out. This can happen when recent traffic is used and Sessions are dropped because they recorded no inputs. For the run size limits, see Start an Optimization Run.

To fix a refused run:

  1. Read the error message and note the Session ids it lists.
  2. On the Operator page, check the run's Left out list. The Session column names each Session, and the Why column gives the reason it was left out.
  3. Remove the Session from the Dataset, or fix the cause (for example, turn on content tracing for the Service so new Sessions record their inputs).
  4. Start the run again. If the run is refused again, repeat these steps for the Sessions the new error message lists.

If you start runs through the API, a refused Dataset run returns a 422 response. Besides errors and test_case_ids, the response lists the affected Sessions in session_ids. Other refusals, such as too many Sessions that couldn't be replayed or too few Sessions left, name each left-out Session by name and id in the error message only. Use these ids to find the Sessions and their traces. To learn where to view traces, see Trace with Built-In Traces.

Fix a Replay That Was Not Recorded​

Use these steps when a baseline run or a candidate fails with a message that says how many replayed turns did not reach search history. The AI Operator replays conversations on the baseline and on each candidate configuration, then judges them. Before it judges a replay, it checks repeatedly that every replayed turn is readable in search history, waiting longer between checks, for about two minutes in total. If some turns still aren't readable when the wait ends, the baseline run or the candidate fails, and nothing is judged from the partial conversation.

To recover:

  1. Check that the Service is running and that its pipeline records traces. Replayed turns can only be judged when they're written to search history.
  2. Start the baseline run again, or request suggestions again, to replay the conversations.
  3. If the failure repeats, open the traces for the Service to see whether recent turns appear. To learn where to view them, see Trace with Built-In Traces.