Part 6 of 12 · 1 min read
What the example found
False-failure and detection rates for the naive and paired gates across six suite sizes.
How to read the table: each row is a different suite size and trial count; the four columns compare the naive (mean-threshold) gate against the paired (sign-test) gate, both when nothing actually changed and when a true 5-point drop was injected, over 600 repeated simulations.
| Suite | Naive gate, nothing changed | Paired gate, nothing changed | Naive gate, true 5-point drop | Paired gate, true 5-point drop |
|---|---|---|---|---|
| 30 cases, 1 run | 45% | 2% | 65% | 7% |
| 30 cases, 3 runs | 37% | 2% | 72% | 17% |
| 100 cases, 1 run | 30% | 3% | 68% | 20% |
| 100 cases, 3 runs | 23% | 4% | 82% | 45% |
| 300 cases, 1 run | 23% | 5% | 82% | 48% |
| 300 cases, 3 runs | 9% | 3% | 94% | 88% |
What to notice: the naive gate's "nothing changed" column never drops below 9%, even at the largest suite tested. It never becomes trustworthy no matter how much evidence you throw at it. The paired gate holds its false-failure rate in the 2-to-5% band at every single suite size. That is the entire reason to prefer it despite its lower detection numbers.
{{svg:flow|A pull request triggers the suite, a paired comparison with the baseline, the gate rules and a report.}} {{svg:results|False failures and detection of a true 5-point drop for a naive and a paired gate, in three suite sizes.}}