Part 3 of 12 · 2 min read
The eval gate: a statistical test, not a score check
A gate that reacts to any observed drop is noisy; one that waits for a confident drop is slow to notice.
Think of two smoke detectors. One is set to go off at the faintest wisp of steam from a shower. The other waits for enough smoke to be sure. The first one annoys everyone constantly. The second one might let a small real fire go unnoticed for a while.
In plain words: an eval gate built on a small set of test cases is comparing two noisy numbers, so a rule that blocks on any observed drop will often block good changes, while a rule that waits for a statistically significant drop will often miss small real regressions.
A worked example. Simulating 1,000 proposed changes, 20 percent of which genuinely regress by 5 points from an 80 percent baseline: with only 200 eval cases, a gate that blocks on any drop over 2 points misses 26 percent of the real regressions and wrongly blocks 34 percent of good changes. A gate that instead requires the drop to be statistically significant blocks far fewer good changes (5 percent) but misses far more real regressions (67 percent). Growing the eval set to 1,000 cases narrows the gap sharply: the significant-drop rule then misses only 14 percent of real regressions while still blocking just 5 percent of good ones.
The key lines.
# from code/mini-deploy/deploy.py
def gate_decision(base_correct, new_correct, n, tolerance=0.02, rule="significant", z_crit=1.645):
p0, p1 = base_correct / n, new_correct / n; drop = p0 - p1
if drop <= tolerance: return False
if rule == "point": return True
se = math.sqrt(p0 * (1 - p0) / n + p1 * (1 - p1) / n) or 1e-9
return drop / se > z_crit
Read it as: the "point" rule blocks as soon as the observed drop passes a small threshold. The "significant" rule additionally asks whether that drop is bigger than the noise expected from the sample size. Neither is free. More eval cases is the lever that lets the significant rule catch more real regressions without blocking more good ones.
CI/CD, quality gate
Continuous integration (CI) automatically builds and tests every change; continuous delivery/deployment (CD) automatically releases the ones that pass. A quality gate is an automated check, like the eval gate above, that blocks a release unless a criterion is met.
Example: the significant-drop rule above running as a required CI check before a prompt change can merge.