CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 2 of 12 · 2 min read

A bare score threshold fails identical builds most of the time

Measuring the same unchanged agent twice almost never gives the same score, so a tolerance band has to be wide enough to absorb that noise, or it blocks good changes constantly.

Think of a bathroom scale that reads half a kilogram differently each morning, even though your actual weight has not moved. A rule that flags "any drop" as a real change would go off every single day for no reason.

In plain words: a regression is a change that makes behaviour worse than before, a baseline is the stored results of the deployed version on the eval set, and a quality gate is an automatic check that passes or fails a change -- but a naive gate that simply compares the candidate's mean score against the baseline's mean score, tolerating only a small drop, fails an unchanged agent constantly, because run-to-run noise in a stochastic suite is often larger than the tolerance.

A worked example. With 30 cases and one run each, a naive gate comparing mean pass rate with a 2-point tolerance fails 45% of builds where the candidate is identical to the baseline -- the agent didn't change at all, but the gate says it did almost half the time. Even with 300 cases and three runs per case, the naive gate still fails 9% of identical builds. An engineering team facing a 45% false-failure rate learns fast to rerun until green, which defeats the entire point of having a gate.

The key lines. From code/mini-gate/gate.py:

# from code/mini-gate/gate.py
def naive_decide(base, cand, tolerance=0.02):
    return {"pass": mean(cand) >= mean(base) - tolerance}

Read it as: the whole decision collapses two sets of per-case numbers into two single means and compares them with one fixed tolerance. Averaging away all the per-case information makes this gate blind to an important difference. It cannot tell "every case drifted a little by chance" from "a few cases actually broke." That is the distinction the next part's gate is built to catch.

Regression, baseline, quality gate

A regression is a change that makes behaviour worse than before. A baseline is the stored results of the currently deployed version on the eval set. A quality gate is an automatic check that passes or fails a change.

Example: the naive gate's 45%-false-failure rate on an identical build is a quality gate that technically runs on every change, but one whose answer you can't trust -- which is the practical difference between "having a gate" and "having a gate worth keeping."