CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 6 of 12 · 2 min read

What the simulation measured

Gate accuracy at different eval-set sizes, rollout damage, and the other mechanical checks.

How to read the gate table: the three eval-set sizes (50 / 200 / 1,000) are read left to right within each cell; a smaller set means a noisier measurement either way.

Gate rule (1,000 changes, 20% really regress by 5 points) Eval cases Real regressions missed Good changes blocked
Any observed drop above 2 points 50 / 200 / 1000 34% / 26% / 6% 43% / 34% / 13%
Only a significant drop 50 / 200 / 1000 85% / 67% / 14% 6% / 5% / 5%
No gate n/a 100% 0%

What to notice: the "significant drop" rule's missed-regression rate falls from 85 percent to 14 percent as the eval set grows from 50 to 1,000 cases. Its good-changes-blocked rate barely moves. A bigger eval set is what buys that rule its accuracy, not a cleverer formula.

How to read the rollout and checks table: the top two rows are the same bad release handled two different ways; the bottom rows are separate, smaller mechanical checks.

Rollout of a bad release (30 min, 50 req/s) Failed requests
All at once, humans notice after 8 min 5,462 (6.1%)
5% canary, automatic rollback after 1 min 936 (1.04%)
Other checks Result
Toy secret scan, 16 lines 7 found, 1 false alarm, 3 missed
Dockerfile lint, bad vs good 4 issues vs 0
Rollback by tag after bad push pulls the bad image
Rollback by digest pulls the previous image
Migrations safe to roll back add nullable column yes; drop, rename, require no

What to notice: the rollout rows show the clearest payoff in the whole module. Automatic canary rollback cuts failed requests by roughly six times, compared with a human noticing within 8 minutes. The underlying bad release is the same.

{{svg:pipeline|Gates from commit to promote, with notes on secrets, images, evals and rollback.}} {{svg:results|Gates, rollout, secrets and rollback in simulation.}}