Pipeline EvaluationBeta
Evaluation turns "the answers seem fine" into numbers you can track. You decide what good looks like for your use case and write it down as a metric. Then you measure your pipeline against it on conversations it already had.
What You Evaluate
You evaluate the conversations and answers your pipeline already produced. This lets you see how your pipeline behaves on real conversations, without having to prepare a test dataset first. To compare two pipeline versions on the same questions, you can also score them on a Dataset of test cases. See Candidate and Baseline Experiments.
A turn is one pipeline run: one query and everything the pipeline did to answer it, including retrieved documents, tool calls, and the final reply. The platform records each turn as a trace.
A session is a conversation: all the turns that share one chat. A single query sent outside a chat counts as a session with one turn. In the Evaluation screens, Traces, and Search History, the platform calls a session a conversation, and this page uses both terms.
Every query you send to your pipeline, from Playground, a deployed app, or the API, becomes a session you can evaluate. Your evaluation reflects how people use the pipeline, not how you expect them to use it.
You evaluate whole sessions, so a metric can judge things a single answer can't show, such as whether the pipeline remembered earlier questions or whether an agent called the right tool.
Main Benefits
Evaluation lets you:
- Check whether a pipeline is ready for production by measuring it on a realistic set of conversations before you share it widely.
- See whether a change helped by measuring before and after you change a prompt, a model, or a retrieval setting.
- Find where the pipeline fails by inspecting the sessions with the lowest scores instead of reading every conversation.
- Catch problems that nobody reports by measuring production conversations.
User feedback answers some of these questions too, but it depends on people taking the time to rate answers. Metrics run on every session you choose, so you get consistent coverage.
Where You Find Evaluation
You can open Evaluation from two places, and both show the same pages:
- Workspace: In the sidebar, under Testing, click Evaluation. The workspace page covers every pipeline in the workspace, so you can see metrics and Datasets in one place.
- Pipeline: Open a pipeline and click Evaluation. The pages show only the content that belongs to that pipeline, for example only the test cases that were added from that pipeline.
Use the workspace page when you work across several pipelines, and the pipeline tab when you focus on one.
If you used Replay Suites before, links to those pages now open Experiments.
How Evaluation Fits Together
Evaluation has four building blocks. Each has its own page:
- Metrics: The questions you ask about every session, answered by an LLM judge or by code. See Evaluation Metrics.
- Datasets: Collections of test cases you keep for measuring pipelines. See Evaluation Datasets and Metric Groups.
- Metric Groups: Named sets of metrics, so you can pick related metrics together instead of one by one. See Evaluation Datasets and Metric Groups.
- Experiments: Runs of a set of metrics over your sessions that you repeat to see whether a change helped, including Candidate and Baseline comparisons. See Evaluation Experiments.
A Measurement, Not a Verdict
Results tell you what happened, not whether it's good enough. There is no built-in pass or fail. You decide what threshold matters for your use case: 0.7 groundedness may be fine for an internal assistant and not good enough for a customer-facing one.
Related Information
Was this page helpful?