Work with AI Operator Suggestions with the REST APIBeta
Compare a suggestion with the baseline in your own tooling, score a suggestion again, or ask for a new round of suggestions without opening the Operator page.
About This Task
All paths on this page are relative to your Service's deployment, except the techniques list, which is relative to the workspace. Each candidate has a proposal_round field: 1 for the run's first round, and one more for each further round.
Get a Suggestion's Results
The response uses the same shape as the results of an Experiment, so you can read both the same way.
-
Send a
GETrequest to this path:/optimization-goals/{goal}/runs/{run}/candidates/{candidate}/results -
Read the response:
metrics: the suggestion's value for each Metric and its delta from the baseline, in the order of the goal's Metric Group.pipeline_metrics: duration, tokens, and cost, aggregated the same way asmetrics.test_cases: one entry for each validation Session the run held out. The baseline side is that Session, or the baseline's replay of it if the Session was recorded on another version. The suggestion side is the suggestion's replay of the Session.status: the comparison's progress:REPLAYING,SCORING,ENDED, orFAILED.pipeline_id,metric_group_id,baseline_experiment_run_id, andcandidate_experiment_run_id: the identifiers you need to open the Sessions and the underlying scoring runs. The two run ids are null until the baseline is queued and the suggestion's replay has finished.
-
The results are partial while the baseline's or the suggestion's evaluation is still running. Poll the endpoint until
statusisENDEDorFAILED. The status isFAILEDwhen either evaluation failed or is gone, or when the suggestion ended without being evaluated.
The endpoint returns a 404 response if the suggestion doesn't exist, belongs to a different run, or if the Service's pipeline no longer exists.
Score a Suggestion Again
If a suggestion's evaluation failed, came back with failed results, or ran out of time, you can score it again without starting a new run. The AI Operator keeps the suggestion's replay and only judges again what didn't succeed.
-
Make sure the suggestion's replay is complete, and that the suggestion failed while it was being scored, finished scoring, was presented, or expired.
-
Send a
POSTrequest to this path:/optimization-goals/{goal}/runs/{run}/candidates/{candidate}/re-evaluate
If the run already ended, it reopens and ends again once the suggestion is scored, then ranks all its suggestions again.
The endpoint returns a 409 response with the code candidate_not_reevaluable in these cases: the suggestion is still being measured, was already accepted, declined, or removed, or its replay is incomplete. It also returns 409 if the run can't be reopened, for example because it isn't the goal's newest run or has an accepted suggestion. It returns 404 if the run or suggestion doesn't exist.
Ask Again
To reopen a run for a new round of suggestions, send a POST request to the run's propose-again endpoint:
curl --request POST \
"https://<API_HOST>/<path-to-the-goal>/runs/<RUN_ID>/propose-again" \
--header "Authorization: Bearer <YOUR_API_KEY>"
The response is the updated run. The request can fail with these status codes:
404: the run doesn't exist.409: the run can't be reopened because it's still in progress, it isn't the goal's newest run, one of its candidates is being promoted or was accepted, or it never measured its baseline.
Rule Out Optimization Techniques
By default, the AI Operator's optimizer is offered every optimization technique. If a technique doesn't suit your Service, for example because you don't want the optimizer to swap the model, rule it out. The optimizer is then told not to propose it. Every suggestion is still validated and measured as usual.
You rule techniques out by their key. The goal stores the techniques you rule out, not the ones you keep, so techniques added later are offered by default.
List the Techniques
To get the keys to rule out, send a GET request to this workspace-level path. It needs read access to Services.
/workspaces/{workspace_id}/optimization-techniques
The response lists the techniques in the order the optimizer reads them. Each entry has a key, a label, a description, and the technique_ids that the optimizer labels its suggestions with. The keys are:
swap_modelreduce_reasoning_effortcompact_historyoptimize_prompt_cachinglimit_agent_stepsimprove_guidance: covers both prompt rewrites (rewrite_prompt) and tool description rewrites (rewrite_tool_description).drop_toolload_tools_on_demandshorten_prompt
Rule Techniques Out for a Goal
To rule techniques out for every run of a goal, set excluded_techniques to a list of keys when you create the goal or when you update it:
curl --request POST \
"https://<API_HOST>/<path-to-the-deployment>/optimization-goals" \
--header "Authorization: Bearer <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{"excluded_techniques": ["swap_model", "drop_tool"]}'
The goal's response includes excluded_techniques.
- On an update, omit the field to keep the current selection. Send
[]to offer every technique again. - A run takes a copy of the goal's selection when it starts. Changing the goal while a run is open doesn't affect that run, including any further round you ask for.
- The request returns a 422 response if a key isn't in the techniques list, or if you rule out every technique. Leave at least one in.
Override the Selection for One Run
To use a different selection for a single run, set excluded_techniques in the request that starts the run. The override replaces the goal's selection for that run only, and the goal isn't changed.
- If you omit the field, the run uses the goal's selection.
- Send
[]to offer every technique in this run, even if the goal rules some out.
The run's response includes the excluded_techniques the run used.
Related Information
Was this page helpful?