Troubleshoot EvaluationBeta
Fix common problems that come up when you evaluate your pipeline with metrics and experiments.
Fix Sessions Where the Judge Didn't Conclude
If a session shows as broke and the run reports that the judge didn't conclude, an LLM judge read the session but never sent a complete, valid report of its answers. Your pipeline, rate limits, and capacity aren't the cause.
An LLM judge gets up to eight steps to read the session and report. If it hasn't sent a valid report by then, it gets two more steps in which it can only report. In that closing round, it's told to answer from what it has read, to use not applicable with a reason instead of guessing, and to fix whatever was wrong with its earlier report. So some sessions can come back as N/A with a reason instead of as broke. A session ends in error only if the judge still hasn't reported after those two steps.
To reduce these errors:
- Open the metric and check that its criterion asks for one specific thing the judge can answer from the session. A criterion that needs a lot of reading uses up the judge's steps before it can report.
- Reduce the number of criteria in the metric, or split it into several metrics. Every criterion needs an answer in the same report.
- For a metric that answers per turn, check how many turns the failing sessions have. A long conversation takes more steps to read. If the failing sessions are long, run the experiment on shorter sessions.
- Use a model that calls tools reliably. A judge reports by calling a tool, so a model that handles tool calls poorly fails more often. In the metric, choose a different model under Which model should judge?.
- Test the metric on the failing sessions with Try, as described in Write and Test an Evaluation Metric, and then run the experiment again. When the run finishes, check the metric's scorecard: fewer sessions show as broke.
Related Information
Was this page helpful?