CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 3 of 12 · 2 min read

Compare per case, not on the average, and ask for statistical evidence

Whether more cases got better or worse than before is a much more stable question than whether the average moved.

Think of a coin you suspect is biased. Flipping it twice and comparing the two outcomes tells you nothing. But counting heads versus tails over a hundred flips lets you make a real statistical statement.

In plain words: a paired comparison compares two versions on the same cases and uses each case's own difference, a sign test counts cases that got better (wins) against cases that got worse (losses) and asks whether that split could plausibly happen by chance -- with pass/fail outcomes this exact test is McNemar's test -- and a false failure is the gate failing a change that didn't actually make anything worse, which this paired approach drives down sharply compared to a naive mean comparison.

A worked example. At 30 cases with one run each, the paired gate's false-failure rate on an identical build is 2%, against the naive gate's 45% -- a more than 20-fold reduction. At 300 cases with three runs, it's 3 to 5% across every suite size tested. The trade for this safety is power: the same paired gate only catches a true 5-point drop 7% of the time at 30 cases with one run, rising to 88% at 300 cases with three runs each -- fewer false alarms costs detection unless the suite is grown to compensate.

The key lines. From code/mini-gate/gate.py:

# from code/mini-gate/gate.py
def paired_sign_test(base, cand):
    """base, cand: {case_id: pass_rate in [0,1]}. Returns (wins, losses, p) over cases whose rate changed."""
    wins = losses = 0
    for k in base:
        d = cand[k] - base[k]
        if d > 1e-12:
            wins += 1
        elif d < -1e-12:
            losses += 1
    return wins, losses, binom_two_sided(min(wins, losses), wins + losses) if wins + losses else 1.0

Read it as: the function only ever looks at each case's own before-and-after difference, never at the overall averages. This removes shared, case-level noise from the comparison. A case that is intrinsically hard in both versions contributes nothing to the win/loss count. So the test is left asking only "did more cases get worse than better?" That is a much steadier question than "did the mean move?"

Paired comparison, sign test, false failure

A paired comparison compares two versions on the same cases and uses each case's own difference. A sign test counts wins and losses and tests whether they're balanced; for pass/fail outcomes this is McNemar's test. A false failure is the gate failing a change that didn't make anything worse.

Example: going from a 45% to a 2% false-failure rate at 30 cases is the sign test doing its one job -- and the module's own honesty line is that this safety isn't free: detection of a real 5-point drop falls to 7% at that same small suite size, so the fix for low power is more cases and more trials, not a looser test.