Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Work with AI Operator Suggestions with the REST APIBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Compare a suggestion with the baseline in your own tooling, score a suggestion again, or ask for a new round of suggestions without opening the Operator page.


About This Task​

All paths on this page are relative to your Service's deployment, except the techniques list, which is relative to the workspace. Each candidate has a proposal_round field: 1 for the run's first round, and one more for each further round.

Get a Suggestion's Results​

The response uses the same shape as the results of an Experiment, so you can read both the same way.

  1. Send a GET request to this path:

    /optimization-goals/{goal}/runs/{run}/candidates/{candidate}/results
  2. Read the response:

    • metrics: the suggestion's value for each Metric and its delta from the baseline, in the order of the goal's Metric Group.
    • pipeline_metrics: duration, tokens, and cost, aggregated the same way as metrics.
    • test_cases: one entry for each validation Session the run held out. The baseline side is that Session, or the baseline's replay of it if the Session was recorded on another version. The suggestion side is the suggestion's replay of the Session.
    • status: the comparison's progress: REPLAYING, SCORING, ENDED, or FAILED.
    • pipeline_id, metric_group_id, baseline_experiment_run_id, and candidate_experiment_run_id: the identifiers you need to open the Sessions and the underlying scoring runs. The two run ids are null until the baseline is queued and the suggestion's replay has finished.
  3. The results are partial while the baseline's or the suggestion's evaluation is still running. Poll the endpoint until status is ENDED or FAILED. The status is FAILED when either evaluation failed or is gone, or when the suggestion ended without being evaluated.

The endpoint returns a 404 response if the suggestion doesn't exist, belongs to a different run, or if the Service's pipeline no longer exists.

Score a Suggestion Again​

If a suggestion's evaluation failed, came back with failed results, or ran out of time, you can score it again without starting a new run. The AI Operator keeps the suggestion's replay and only judges again what didn't succeed.

  1. Make sure the suggestion's replay is complete, and that the suggestion failed while it was being scored, finished scoring, was presented, or expired.

  2. Send a POST request to this path:

    /optimization-goals/{goal}/runs/{run}/candidates/{candidate}/re-evaluate

If the run already ended, it reopens and ends again once the suggestion is scored, then ranks all its suggestions again.

The endpoint returns a 409 response with the code candidate_not_reevaluable in these cases: the suggestion is still being measured, was already accepted, declined, or removed, or its replay is incomplete. It also returns 409 if the run can't be reopened, for example because it isn't the goal's newest run or has an accepted suggestion. It returns 404 if the run or suggestion doesn't exist.

Ask Again​

To reopen a run for a new round of suggestions, send a POST request to the run's propose-again endpoint:

curl --request POST \
"https://<API_HOST>/<path-to-the-goal>/runs/<RUN_ID>/propose-again" \
--header "Authorization: Bearer <YOUR_API_KEY>"

The response is the updated run. The request can fail with these status codes:

  • 404: the run doesn't exist.
  • 409: the run can't be reopened because it's still in progress, it isn't the goal's newest run, one of its candidates is being promoted or was accepted, or it never measured its baseline.

Rule Out Optimization Techniques​

By default, the AI Operator's optimizer is offered every optimization technique. If a technique doesn't suit your Service, for example because you don't want the optimizer to swap the model, rule it out. The optimizer is then told not to propose it. Every suggestion is still validated and measured as usual.

You rule techniques out by their key. The goal stores the techniques you rule out, not the ones you keep, so techniques added later are offered by default.

List the Techniques​

To get the keys to rule out, send a GET request to this workspace-level path. It needs read access to Services.

/workspaces/{workspace_id}/optimization-techniques

The response lists the techniques in the order the optimizer reads them. Each entry has a key, a label, a description, and the technique_ids that the optimizer labels its suggestions with. The keys are:

  • swap_model
  • reduce_reasoning_effort
  • compact_history
  • optimize_prompt_caching
  • limit_agent_steps
  • improve_guidance: covers both prompt rewrites (rewrite_prompt) and tool description rewrites (rewrite_tool_description).
  • drop_tool
  • load_tools_on_demand
  • shorten_prompt

Rule Techniques Out for a Goal​

To rule techniques out for every run of a goal, set excluded_techniques to a list of keys when you create the goal or when you update it:

curl --request POST \
"https://<API_HOST>/<path-to-the-deployment>/optimization-goals" \
--header "Authorization: Bearer <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{"excluded_techniques": ["swap_model", "drop_tool"]}'

The goal's response includes excluded_techniques.

  • On an update, omit the field to keep the current selection. Send [] to offer every technique again.
  • A run takes a copy of the goal's selection when it starts. Changing the goal while a run is open doesn't affect that run, including any further round you ask for.
  • The request returns a 422 response if a key isn't in the techniques list, or if you rule out every technique. Leave at least one in.

Override the Selection for One Run​

To use a different selection for a single run, set excluded_techniques in the request that starts the run. The override replaces the goal's selection for that run only, and the goal isn't changed.

  • If you omit the field, the run uses the goal's selection.
  • Send [] to offer every technique in this run, even if the goal rules some out.

The run's response includes the excluded_techniques the run used.