Part 6 of 12 · 2 min read
What the simulation measured
Alert rules on simulated traffic, a cost alert, and the cost of each mitigation choice.
How to read the alerting table: the two rows are the same two rules facing three separate scenarios; "false pages" is the cost of a rule, the other two columns are its benefit.
| Rule on simulated traffic (assumed) | Sudden outage, minutes to page | Slow burn, minutes to page | False pages in 7 quiet days |
|---|---|---|---|
| 1-minute error rate above 5% | 0 | 24 (by chance) | 44 |
| Multi-window burn rate | 5 | 75 | 2 |
What to notice: the threshold rule's "24 minutes" on the slow burn looks like a win over burn rate's 75. But it is a coincidence of noise crossing 5% at that moment, not a repeatable detection. The real trade-off is burn rate's 5-minute slower response to a sudden outage against 42 fewer false pages a week.
A cost-rate alert works the same way for spend. Take a runaway loop raising spend from $0.20 to $3.00 per minute against a $30-per-hour budget, checked over a 15-minute window. It produces its first page at minute 301. That is one minute after the spike starts. The window-averaged rate crosses the hourly budget almost immediately once the spike is large enough.
How to read the mitigation table: all four rows face the identical bug (5 bad actions per minute, detected at minute 8); the trade-off is in the right two columns, not the left.
| Mitigation (5 bad actions per minute, detected at minute 8) | Minutes until harm stops | Bad actions | Legitimate actions blocked |
|---|---|---|---|
| Wait for a fix (60 min) | 68 | 340 | 0 |
| Roll back the release (15 min) | 23 | 115 | 0 |
| Kill switch, left on 60 min | 10 | 50 | 720 |
| Kill switch, lifted at rollback (15 min) | 10 | 50 | 156 |
What to notice: the last row dominates every other option on both bad actions and collateral at once. It pairs the fast switch with the rollback, then lifts the switch the moment the rollback lands. That gets the low bad-action count of a kill switch without paying for 720 minutes of blocked legitimate work.
{{svg:lifecycle|The incident lifecycle with the three clocks, and the rules that keep response fast.}} {{svg:results|Alert rules on simulated traffic, and the cost of each mitigation.}}