Evaluation Goal Checks in Deployment GatesBeta
A goal check lets a Deployment Gate judge a new revision by the Evaluation goals of the Service's pipeline, so one set of goals answers whether the pipeline is good enough to ship. Use it when you already define quality as goals and don't want to maintain a separate bar on a Metric Group's Composite. The revision goes live only when every listed goal is met.
What a Goal Check Contains
A goal check has these parts:
- Evaluation goals: The goals of the Service's pipeline that the candidate must meet. List each goal once.
- Dataset: The Dataset of test cases to replay through the new revision.
- Minimum coverage: The share of replayed test cases, from 0 to 1, that must be judged for every goal's Metric. Below it the check is undecidable. The default is 0.8.
A gate can have one goal check. You can't set up goal checks in the UI or the public API yet. Support is coming.
How Goals Are Frozen
A goal check follows your goals as they are. Each time the platform holds a revision, it freezes the listed goals together with the latest version of each goal's Evaluator, and it judges the revision on those frozen goals. If you edit a goal afterward, the verdict of that run doesn't change. A goal you deleted before the revision was held is left out.
To judge the goals, the platform scores the replayed Sessions on the pipeline's goal group, the Metric Group that holds the pipeline's Evaluation goals. You don't pick this group. For how goals decide whether a session passes, see Evaluation Goals.
Retrying an undecidable result freezes the goals again from their current state. Retrying a Composite check repeats the same run.
When a Goal Check Is Undecidable
A goal check is undecidable in these cases, besides the general causes such as a failed Scoring Run, a failed replay, low coverage, or an Evaluator version the check didn't pin:
- All of its goals were deleted before the revision was held.
- Its goals don't belong to the pipeline of the held revision.
- It would need more evaluations than a single Scoring Run allows.
- A goal's Evaluator has no version to judge the goal with.
- A goal has no Session to judge it by.
These cases come from the goals, the Dataset, or the Evaluators, not from the revision's quality. The previous revision keeps serving. Fix the cause, then retry the result: re-add the pipeline's goals, use a smaller Dataset, or make sure each goal's Evaluator has a version and the Dataset produces Sessions for every goal. Retrying freezes the goals again from their current state.
Related Information
Was this page helpful?