Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Work with AI Operator Suggestions Through the APIBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Compare a suggestion with the baseline in your own tooling, score a suggestion again, or ask for a new round of suggestions without opening the Operator page.

About This Task​

All paths on this page are relative to your Service's deployment. Each candidate has a proposal_round field: 1 for the run's first round, and one more for each further round.

Get a Suggestion's Results Through the API​

The response uses the same shape as the results of an Experiment, so you can read both the same way.

  1. Send a GET request to this path:

    /optimization-goals/{goal}/runs/{run}/candidates/{candidate}/results

  2. Read the response:

    • metrics: the suggestion's value for each Metric and its delta from the baseline, in the order of the goal's Metric Group.
    • pipeline_metrics: duration, tokens, and cost, aggregated the same way as metrics.
    • test_cases: one entry for each validation Session the run held out. The baseline side is that Session, or the baseline's replay of it if the Session was recorded on another version. The suggestion side is the suggestion's replay of the Session.
    • status: the comparison's progress: REPLAYING, SCORING, ENDED, or FAILED.
    • pipeline_id, metric_group_id, baseline_experiment_run_id, and candidate_experiment_run_id: the identifiers you need to open the Sessions and the underlying scoring runs. The two run ids are null until the baseline is queued and the suggestion's replay has finished.

  3. The results are partial while the baseline's or the suggestion's evaluation is still running. Poll the endpoint until status is ENDED or FAILED. The status is FAILED when either evaluation failed or is gone, or when the suggestion ended without being evaluated.

The endpoint returns a 404 response if the suggestion doesn't exist, belongs to a different run, or if the Service's pipeline no longer exists.

Score a Suggestion Again Through the API​

If a suggestion's evaluation failed, came back with failed results, or ran out of time, you can score it again without starting a new run. The AI Operator keeps the suggestion's replay and only judges again what didn't succeed.

  1. Make sure the suggestion's replay is complete, and that the suggestion failed while it was being scored, finished scoring, was presented, or expired.

  2. Send a POST request to this path:

    /optimization-goals/{goal}/runs/{run}/candidates/{candidate}/re-evaluate

If the run already ended, it reopens and ends again once the suggestion is scored, then ranks all its suggestions again.

The endpoint returns a 409 response with the code candidate_not_reevaluable in these cases: the suggestion is still being measured, was already accepted, declined, or removed, or its replay is incomplete. It also returns 409 if the run can't be reopened, for example because it isn't the goal's newest run or has an accepted suggestion. It returns 404 if the run or suggestion doesn't exist.

Ask Again Through the API​

To reopen a run for a new round of suggestions, send a POST request to the run's propose-again endpoint:

curl --request POST \
  "https://<API_HOST>/<path-to-the-goal>/runs/<RUN_ID>/propose-again" \
  --header "Authorization: Bearer <YOUR_API_KEY>"

The response is the updated run. The request can fail with these status codes:

  • 404: the run doesn't exist.
  • 409: the run can't be reopened because it's still in progress, it isn't the goal's newest run, one of its candidates is being promoted or was accepted, or it never measured its baseline.