Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Monitor LLM Costs and Token UsageBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

To keep track of your LLM spend and find out which workspaces, services, pipelines, providers, and models drive it, use the Cost Management page. It shows how much your LLM calls cost and how many tokens they use.

About This Task​

The Cost Management page shows costs for your whole organization or, depending on your role, for a workspace. Costs are estimates: we calculate them from the token usage recorded in your traces and the model's price. To check credits and document storage units instead, see Understand Your Pipeline Usage. To change a model's price, see Set Model Prices.

The page refreshes automatically every minute while it's open. Click the refresh indicator in the filter bar to refresh right away.

Permissions

Organization Admins can see the cost data of the whole organization and of any workspace. Other users can open the page if their role has the Costs permission. They see the data of the workspace they're in and can't switch to another one. For details, see User Roles and Permissions.

Open the Cost Management Page​

In Haystack Enterprise Platform, click Cost Management in the sidebar, or open the command palette and search for Cost Management. You can see the spend for the whole organization over the date range you choose.

The page URL keeps the scope, date range, and filters, so to share your current view, copy the URL and send it.

Read the Summary​

To check how much you've spent and where the month is heading, read the summary at the top of the page. It shows these figures for the scope and date range you selected:

  • Total spend: The cost of all priced calls in the range, with a comparison against the previous period of the same length. If there is no earlier data, the page says No prior period to compare yet.
  • Requests: The number of LLM calls.
  • Tokens: The number of tokens used by those calls.
  • Avg / request: The average cost of one call.
  • Avg / day: The average cost per day in the range.
  • Projected this month: The month-to-date spend plus the average daily spend of the last 7 days applied to the remaining days of the month. Use it to see where the month is heading.

Calls That Aren't in the Total​

If some calls couldn't be priced yet, or can't be priced at all, a warning appears under the summary.

  • Calls that are still being priced are added automatically, usually within seconds.
  • Calls to models without a matching price appear under Unmatched models. If you're an Organization Admin, click a model name to set a price for it, which applies to all of its calls. For details, see Set Model Prices.

Explore the Breakdown​

To compare spend across groups for the whole date range, use the Spend breakdown table below the Cost flow chart. It lists spend by the dimension you choose in its Group by control. The options depend on your scope:

  • Whole organization: Workspace, Service, Pipeline, Model, and Provider.
  • One workspace: Service, Pipeline, Model, and Provider.
  • One service or pipeline: Model and Provider. The table groups by provider by default.

Each row shows:

  • Share of spend: The row's share of the total.
  • Requests: The number of calls.
  • Tokens: The number of tokens.
  • Avg / req: The average cost of one call.
  • Cost: The row's total cost.

Click a workspace, service, or pipeline row to scope the whole page to it. Click a provider row to filter the page by that provider.

To see which providers take the largest share of spend, check the By provider card. To find your most expensive models, check the Top models card, which lists the five models with the highest cost. To follow spend from your organization through workspaces and services to models, use the Cost flow chart. It shows which workspaces and services send the most spend to which models. Hover over a flow to see its cost. The Services column covers both services and pipelines. When you narrow to one workspace, the chart starts at the workspaces column.

Get Daily Cost Over the API​

To use daily cost data outside the Cost Management page, for example in your own reports, call the cost-over-time endpoint. It returns the cost per UTC day in US dollars:

GET /cost/organizations/{organization_id}/stats/cost-over-time

Users with the Costs permission must pass a workspace_id. Only Organization Admins can leave it out to get organization-wide data.

Without extra parameters, the response contains one point per day. To split each day, add these optional query parameters:

  • group_by: The dimension to split by: model, workspace, pipeline, deployment, provider, or api_key. deployment is called a service in the UI.
  • group_limit: How many of the groups with the highest cost keep their own points. The default is 5 and the maximum is 20.
curl --request GET \
  "$API_BASE_URL/cost/organizations/$ORGANIZATION_ID/stats/cost-over-time?group_by=provider&group_limit=5" \
  --header "Authorization: Bearer $API_KEY"

With group_by, the response contains:

  • group_by: The dimension you chose.
  • groups: Every group the points refer to, in stack order. Each group has a key, a label, and a kind. The kind is group for a group with high cost, other for the merged remaining groups, or unattributed for usage with no value for the dimension.
  • data: One point per day and group. Each point has a group field with one of the keys in groups. A day's points add up to the day's total.

Without group_by, group_by and groups are null.

The API also has endpoints for totals, breakdown, and cost flow. The breakdown endpoint returns one page of groups at a time. Use sort_by, sort_order, limit, and page_number to sort the groups and page through them, and check has_more to see whether more pages follow.