Set Up an AI Operator for a ServiceBeta

Beta

This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Add an AI Operator to a running Service to measure what it costs and how well it answers, then ask it for cheaper configurations.

---

About This Task

Before you start, make sure you have:

• A running Service you want to optimize.
• An Experiment for that Service's pipeline, with evaluators that judge answer quality. If the pipeline has no Experiment yet, the Add AI Operator modal tells you so and links to where you can create one.

Each Service can have only one AI Operator. If a Service already has a goal, it still appears in the Add AI Operator list, but it's disabled.

Add an AI Operator

1. Go to Operate > AI Operator, or open a Service and click Set up AI Operator next to the actions menu.
2. Click Add AI Operator.
3. Select the running Service you want to optimize.
4. Select the Experiment whose evaluators judge this Service's answers.
5. Set the quality the Service must keep while being optimized.
6. Click Save to create the goal.

Result: The Service now has an AI Operator. Its row appears on the Operator list with a status based on its latest run, and the Service page shows an AI Operator button in place of Set up AI Operator.

View the Operator List

To check which Services are being optimized, whether each has a baseline, and when each was last measured, go to Operate > AI Operator. The list shows every optimization goal in the workspace. For each row, you see:

• Service: the Service the goal belongs to.
• Goal: the goal written as a sentence, so you can read what it keeps and what it optimizes for.
• Status: read from the latest measurement run (for example, whether a baseline exists or a run is in progress).
• Last run: when the Operator last measured the Service.

Click a row to open that Operator's page.

Measure a Baseline

Measure a baseline to record the Service's current cost, latency, and quality. You need a baseline before you can request suggestions, and it gives you numbers to compare them against.

1. Open the Operator page for the Service, either from the Operator list or the Service page.
2. Choose the pipeline version you want to measure.
3. Start the measurement run.
4. Review the baseline once the run finishes: cost, latency, and quality, along with the conversations the run used.

Review Suggestions

Once a baseline exists, you can ask the AI Operator for optimization suggestions:

1. On the Operator page, request suggestions.
2. Review each suggested configuration together with its validated cost and quality.
3. Copy a suggestion's configuration if you want to try it yourself. The AI Operator does not apply or promote any suggestion automatically — you decide what to do with it.

Edit or Remove the Goal

On the Operator page, you can:

• Edit the goal to change the Experiment or the quality threshold.
• Remove the goal, which also removes the Service from the Operator list.

What To Do Next

• AI Operator explains the concepts behind goals, baselines, and suggestions.
• Run an Experiment to learn more about the evaluators an AI Operator relies on.
• View Cost and Quality Trends on the AI Operator Page to see how the Service's cost and quality change across releases.