Part 4 of 12 · 2 min read
A critical case gets zero tolerance; everything else gets a budget
An improved average can hide a single new catastrophic failure, so some cases have to be checked on their own, outside the statistics.
Think of an aircraft pre-flight checklist. Most items get judged on an overall "is the plane airworthy" assessment. But a few items fail the check on their own, such as a cracked engine blade or a jammed control surface. They fail it regardless of how good everything else looks.
In plain words: a critical case is a case whose failure is unacceptable regardless of the average, so it's checked with zero tolerance rather than folded into the statistical test, and the gate can combine this absolute check with a separate cost-ratio limit, so a candidate that's statistically fine on quality can still be blocked for being, say, 50% more expensive than the baseline on the same cases.
A worked example. In decide(), every name listed in critical is checked for a pass rate of exactly 1.0 -- not "close to," not "within the noise band," but 1.0 -- so a single critical case that now fails even once out of its trials is an automatic build failure, independent of whatever the paired sign test concludes about the rest of the suite. This matters because an average can rise while one case silently breaks: a quality gate that only looked at the mean could pass a build where a money-handling or safety case had gone from reliable to broken.
The key lines. From code/mini-gate/gate.py:
# from code/mini-gate/gate.py
def decide(base, cand, alpha=0.1, min_drop=0.02, critical=(), max_cost_ratio=None, cost_base=None, cost_cand=None):
"""Return {'pass': bool, 'reasons': [...]}. A regression needs BOTH a real drop and statistical evidence."""
reasons = []
for c in critical:
if cand.get(c, 0.0) < 1.0:
reasons.append(f"critical case {c} failed")
Read it as: the critical-case check runs first, as its own loop, completely separate from the statistical drop test that follows it. This ordering is the point. A critical case failing adds its own reason to block the build, regardless of what the rest of the function decides. So there is no way for a good average to outvote one broken critical case.
Critical case
A critical case is a case whose failure is unacceptable regardless of the average.
Example: a refund-handling case dropping from "always correct" to "correct most of the time" would barely move a 300-case average, but listing it as critical turns that same drop into an automatic, non-negotiable build failure -- which is exactly the gap a pure statistical test can't close on its own.