Part 9 of 12 · 2 min read
Mistakes that turn an incident into a worse one
Common mistakes, and what the example does not do.
Mistakes that turn an incident into a worse one
- Diagnosing before mitigating.
- No kill switch, or one that needs a deploy.
- Kill switches that fail open or have no audit trail.
- Alerting on every spike, leading to ignored pages.
- Runbooks that say "investigate" or were never drilled.
- Treating wrong actions like downtime: forgetting to find and repair what the agent already did.
- Deleting logs or traces while cleaning up.
- Blameful post-mortems, so people hide information next time.
- One root cause only and no actions that improve detection.
- Actions without owners or dates.
What the example does not do
About this example What the example does not do
- No real paging, dashboard or on-call process; only alert rules over a simulated series.
- The noise model is invented: real error series are bursty, seasonal and correlated, so false-page counts will differ.
- Burn-rate windows and factors are common choices, not tuned; tune them to your traffic and tolerance.
- The harm simulation is a straight line: real incidents have ramps, partial fixes and side effects.
- Severity rules are illustrative, not an industry standard; the legal and contractual notification duties are not modelled and need counsel.
- The post-mortem checker only tests structure and a short list of blame words; it cannot judge quality.
- Optional real experiment: hold a one-hour game day on your own system: inject a bad prompt in staging, time detection, flip a kill switch and roll back, then write the post-mortem and run
check_postmortemon it.
Check yourself
A refund bug sends 5 wrong refunds a minute and was detected after 8 minutes. Compare waiting for a fix (60 minutes), rolling back (15) and a kill switch (2).
Show answer
340, 115 and 50 bad actions. The kill switch stops the harm fastest, at the cost of blocking legitimate refunds while it is on.
For the curious Alert on burn rate, not a single noisy minute
A static 1-minute error threshold is fast but noisy. A multi-window burn-rate rule pages only when the error budget burns fast over both a short and a long window. The series are simulated at 200 requests a minute with assumed noise. A runbook that says investigate has no mitigation: each needs an owner, a concrete switch, flag or rollback, an escalation path and a recent drill.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Mitigate first, diagnose second | Disable a tool, fall back, require approval, pause a tenant, or stop. |
| Keep scoped kill switches with an audit trail | Who, why and when. |
| Write the runbook before the incident | Owner, first check, mitigation, rollback, escalation, last drill. |
| Hold a blameless post-mortem | Plural root causes and owned action items. |