CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 6 of 12 · 1 min read

What the example found

False-failure and detection rates for the naive and paired gates across six suite sizes.

How to read the table: each row is a different suite size and trial count; the four columns compare the naive (mean-threshold) gate against the paired (sign-test) gate, both when nothing actually changed and when a true 5-point drop was injected, over 600 repeated simulations.

Suite Naive gate, nothing changed Paired gate, nothing changed Naive gate, true 5-point drop Paired gate, true 5-point drop
30 cases, 1 run 45% 2% 65% 7%
30 cases, 3 runs 37% 2% 72% 17%
100 cases, 1 run 30% 3% 68% 20%
100 cases, 3 runs 23% 4% 82% 45%
300 cases, 1 run 23% 5% 82% 48%
300 cases, 3 runs 9% 3% 94% 88%

What to notice: the naive gate's "nothing changed" column never drops below 9%, even at the largest suite tested. It never becomes trustworthy no matter how much evidence you throw at it. The paired gate holds its false-failure rate in the 2-to-5% band at every single suite size. That is the entire reason to prefer it despite its lower detection numbers.

{{svg:flow|A pull request triggers the suite, a paired comparison with the baseline, the gate rules and a report.}} {{svg:results|False failures and detection of a true 5-point drop for a naive and a paired gate, in three suite sizes.}}