Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Add Suggested Budget and Response Time GoalsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Use this when you start evaluating a pipeline and want a first pair of goals, cost and response time, in a few clicks. Cost and response time are limits that almost every pipeline needs, and neither needs an LLM judge, so the empty Evaluation tab proposes them directly.


About This Task​

When a pipeline has no Evaluation goals, the Evaluation tab lists two suggested goals under We suggest starting with:

  • Budget (Cost category): each turn costs at most a dollar amount in a share of sessions. The default is $1 in 100% of sessions. It uses the built-in Max turn cost Evaluator template.
  • Response time (Efficiency category): each answer arrives within a number of seconds in a share of sessions. The default is 20 seconds in 100% of sessions. It uses the built-in Max turn duration Evaluator template, which measures the slowest turn of a session. A session is not applicable if any of its turns has no recorded duration, so unmeasured waiting never passes the goal.

If your workspace already has an Evaluator with the template's name, the goal uses it. Otherwise, the platform creates the Evaluator from the template when you add the goal.

The tab also offers Something about quality?, which opens the New Evaluation goal page. When the AI Assistant is available, it helps you define a goal for quality there. To use it, see Start with Suggested Evaluation Goals.

Add the Suggested Goals​

Before you start, make sure the pipeline has no Evaluation goals yet. Otherwise, the tab shows your goals and no suggestions.

  1. Open your pipeline and click Evaluation.
  2. In the We suggest starting with list, change the threshold (the dollar amount or the number of seconds) and the pass rate if the defaults don't fit your pipeline.
  3. Click Add on a suggestion to create its goal, or click Add both goals to create both.
  4. If you added the goals one at a time, click Done to open the list of your goals. Add both goals opens the list on its own.
  5. To score the goals on your live sessions, click Evaluate now. Scoring runs in the background. When it finishes, each goal in the list shows whether it is met. If a goal isn't met, adjust its threshold if it doesn't fit your pipeline.

After a goal exists, you can edit it like any other goal, including its aggregation. For the fields, see Define Evaluation Goals on a Pipeline.

Troubleshooting​

  • A suggestion is missing. A suggestion appears only when its Evaluator template is available in your environment. If only the Budget suggestion appears, the Max turn duration template isn't available yet. Click New Evaluation goal to add a goal manually in the meantime.
  • Only one goal exists after Add both goals. Add both goals saves the goals one after the other. If the second save fails, no error appears and the list lacks that goal. Click New Evaluation goal to create the missing goal.