View Cost and Quality Trends on the AI Operator PageBeta
To find out whether a release changed what a Service costs and how well it answers, use the Service Cost History card on the Service's AI Operator page. It shows daily cost next to the published revisions and the measured quality of each evaluation run.
About This Task
You see the Service Cost History card only if you have permission to read costs. If you don't see it, ask your workspace admin for access. The Service also needs an AI Operator; see Set Up an AI Operator for a Service.
The card sits above the run cards, next to the Runs table. Its header shows today's cost per conversation, so you can read the current cost without checking the chart. Start with the trends across all runs there, then look at the run cards below for the details of the selected run. To learn about the Runs table, see View Past Runs.
Chart Elements
The card includes:
- Cost per day: a line showing the Service's total cost for each day in the selected period. This is the Service's total spend, so it also rises and falls with traffic, not only with configuration changes.
- Version markers: a dashed vertical line for each published revision that served traffic, labeled with its version. Revisions that failed to deploy are left out, so every marker reflects a release your users could reach.
- Quality (average score) dots: a dot for each evaluation run, plotted on a second axis from 0 to 1. Each dot is the run's baseline quality, calculated as the mean of its scored metrics. Label metrics and evaluations that haven't finished are skipped, so a dot always reflects a completed, scored run.
- Date range selector: switch between 30 days, 90 days, and 6 months to widen or narrow the window shown in the chart.
If the card shows no cost for the selected period, select a longer date range. If the cost data doesn't load, refresh the page.
Read the Chart
- Open the Service and go to its AI Operator page.
- Find the Service Cost History card.
- Select a date range that covers the releases you want to compare.
- Compare the cost line against the dashed version markers to see whether a cost change lines up with a specific release.
- Compare the quality dots near a version marker to see whether that release changed answer quality, not just cost. Because cost also moves with traffic, use the markers and dots together to judge whether a release caused a cost change. If a release raised cost without improving quality, start an optimization run; see Start an Optimization Run.
See What Each Optimization Run Cost
The Service Cost History card shows your Service's total spend. To see what the AI Operator itself spent on each run of a goal, such as replaying your dataset, evaluating answers, and generating suggestions, open the run costs page. Use it to check how much an optimization run cost and which phase accounts for most of it.
- Open the Service and go to its AI Operator page.
- In the goal bar, click View costs. The button appears once the goal has at least one run.
- Check the summary card at the top for the total cost of the runs on the page and the total for each phase.
- Read the table below it. Runs are listed newest first, and each row shows when the run started, its result, the baseline version, the cost of each phase, and the run's total. Hover over the time to see the exact date and time, and over a cost to see the number of LLM calls and the input and output tokens behind it.
- To see what each suggestion cost, expand a run. Suggestions carry the same names as their suggestion cards, for example Swap model. For each suggestion, you see the cost of replaying it and of evaluating it.
- To go further back, click Show older runs.
The cost phases are:
- Baseline replays: running your dataset through the current pipeline version.
- Baseline evaluation: scoring the baseline answers.
- Optimizer: generating suggestions.
- Suggestion replays: running your dataset through each suggestion.
- Suggestion evaluation: scoring the answers each suggestion gives.
Cost States
A cost cell doesn't always show a number:
—means the step didn't run for that run.- Pricing means some calls are still being priced. The total for that run stays blank until pricing finishes, and the page refreshes every 10 seconds while the tab is in focus until it does. It also refreshes while a run is still in progress.
- A warning
—means some models have no price. Hover over it to see which models. To fix it, set a price for them. For details, see Set Model Prices. - Not recorded means the run took place before the AI Operator recorded its spend, so no cost is available.
The summary leaves out runs whose total isn't available yet, instead of counting them as $0, and tells you how many it left out. When the pricing finishes, those runs are included in the summary.
What an Optimization Run's Cost Includes
An optimization run's cost covers everything the run set off, counted at the moment it started. Use this when a run's total looks higher or lower than you expect.
- Repeated work counts. If a suggestion is replayed again, or the baseline or a suggestion is evaluated again, the earlier attempts stay in the cost. Replayed turns are counted one by one, so a test case that stopped partway still counts the turns that ran. An optimizer call that returned no suggestions also counts.
- Reused work isn't counted twice. When a run reuses baseline replays or earlier evaluation results, the run that paid for them first carries their cost.
- Recent turns can appear late. A replayed turn is counted once its token usage is written, which usually takes a few seconds.
- Optimizer calls are billed to the optimizer's workspace. Its cost still appears in the run's total.
To read the same breakdown in your own tooling or scripts, call the run's spend endpoint, GET /api/v2/workspaces/{workspace_id}/deployments/{deployment_id}/optimization-goals/{goal_id}/runs/{run_id}/spend. It returns the total, the cost of each of the five phases, and the cost of each suggestion's replay and evaluation. For a run that started before spend was recorded, the total is null because the cost is unknown, not zero.
Related Information
Was this page helpful?