Part 6 of 12 · 2 min read
What the simulation measured
Gate accuracy at different eval-set sizes, rollout damage, and the other mechanical checks.
How to read the gate table: the three eval-set sizes (50 / 200 / 1,000) are read left to right within each cell; a smaller set means a noisier measurement either way.
| Gate rule (1,000 changes, 20% really regress by 5 points) | Eval cases | Real regressions missed | Good changes blocked |
|---|---|---|---|
| Any observed drop above 2 points | 50 / 200 / 1000 | 34% / 26% / 6% | 43% / 34% / 13% |
| Only a significant drop | 50 / 200 / 1000 | 85% / 67% / 14% | 6% / 5% / 5% |
| No gate | n/a | 100% | 0% |
What to notice: the "significant drop" rule's missed-regression rate falls from 85 percent to 14 percent as the eval set grows from 50 to 1,000 cases. Its good-changes-blocked rate barely moves. A bigger eval set is what buys that rule its accuracy, not a cleverer formula.
How to read the rollout and checks table: the top two rows are the same bad release handled two different ways; the bottom rows are separate, smaller mechanical checks.
| Rollout of a bad release (30 min, 50 req/s) | Failed requests |
|---|---|
| All at once, humans notice after 8 min | 5,462 (6.1%) |
| 5% canary, automatic rollback after 1 min | 936 (1.04%) |
| Other checks | Result |
|---|---|
| Toy secret scan, 16 lines | 7 found, 1 false alarm, 3 missed |
| Dockerfile lint, bad vs good | 4 issues vs 0 |
| Rollback by tag after bad push | pulls the bad image |
| Rollback by digest | pulls the previous image |
| Migrations safe to roll back | add nullable column yes; drop, rename, require no |
What to notice: the rollout rows show the clearest payoff in the whole module. Automatic canary rollback cuts failed requests by roughly six times, compared with a human noticing within 8 minutes. The underlying bad release is the same.
{{svg:pipeline|Gates from commit to promote, with notes on secrets, images, evals and rollback.}} {{svg:results|Gates, rollout, secrets and rollback in simulation.}}