Definitions

Definitions

The lifecycle and the controls

Results

1. Burn rate trades a few minutes of delay for far fewer false pages, and it is the one that notices slow degradation reliably.
2. A narrow kill switch plus a rollback gave the least total harm in this scenario.
3. The threshold rule's 24-minute slow-burn catch was luck, not detection; do not read it as a strength.

Design rules

• Page on user-visible symptoms (errors, latency, wrong actions, cost) rather than on every internal cause.
• Use burn-rate alerts for availability and quality signals; keep a simple threshold for hard limits such as spend.
• Every page needs a runbook and an owner; delete pages nobody acts on.
• Build the kill switches before you need them: per tool, per tenant, global; fail closed; logged with who and why; tested monthly.
• Make rollback one command and rehearse it (module 8, module 9).
• Separate roles in a serious incident: commander, fixer, communicator.
• Communicate: status updates at a fixed cadence; for customer data incidents, follow your notification duties (Level 10 module 10) and ask counsel, since this module is not legal advice.
• Preserve evidence: traces, journals and the audit chain (Level 10 module 9); do not delete logs while mitigating.
• Write the post-mortem within days, blameless, with owned and dated actions; track them to completion.
• Turn every incident into an eval, an alert or a control.

Interview questions, with the follow-ups

Common mistakes

• Diagnosing before mitigating.
• No kill switch, or one that needs a deploy.
• Kill switches that fail open or have no audit trail.
• Alerting on every spike, leading to ignored pages.
• Runbooks that say "investigate" or were never drilled.
• Treating wrong actions like downtime: forgetting to find and repair what the agent already did.
• Deleting logs or traces while cleaning up.
• Blameful post-mortems, so people hide information next time.
• One root cause only and no actions that improve detection.
• Actions without owners or dates.

What the example does not do

• No real paging, dashboard or on-call process; only alert rules over a simulated series.
• The noise model is mine: real error series are bursty, seasonal and correlated, so false-page counts will differ.
• Burn-rate windows and factors are common choices, not tuned; tune them to your traffic and tolerance.
• The harm simulation is a straight line: real incidents have ramps, partial fixes and side effects.
• Severity rules are illustrative, not an industry standard; the legal and contractual notification duties are not modelled and need counsel.
• The post-mortem checker only tests structure and a short list of blame words; it cannot judge quality.
• Optional real experiment (I did not run this): hold a one-hour game day on your own system: inject a bad prompt in staging, time detection, flip a kill switch and roll back, then write the post-mortem and run check_postmortem on it.

For the curious Alert on burn rate, not a single noisy minute

A static 1-minute error threshold is fast but noisy. A multi-window burn-rate rule pages only when the error budget burns fast over both a short and a long window. The series are simulated at 200 requests a minute with assumed noise. A runbook that says investigate has no mitigation: each needs an owner, a concrete switch, flag or rollback, an escalation path and a recent drill.

What you must be able to say (one page)

• An incident is any event that harms users or the business and needs a coordinated response. For agents the usual causes are a bad release (prompt, model or tool change), a provider fault, a runaway loop or cost spike, a cross-tenant leak, a prompt-injection exploit, or wrong actions (refunds, emails) at scale. The lifecycle is start, detect, acknowledge, mitigate, resolve, learn; the clocks time to detect, time to mitigate and time to resolve are the numbers you improve.
• Mitigate first, diagnose second. Stopping the harm beats understanding it. The mitigation ladder, from narrowest to widest: disable one tool, switch to a fallback model, require human approval for all writes, pause one tenant, global stop. Roll back a release (module 9) when a release caused it. Kill switches must be scoped, fail-closed, logged (who and why) and practised.
• Alert on symptoms users feel, with rules that resist noise. A static "1-minute error rate above 5%" rule is fast but noisy; a multi-window burn-rate rule pages only when the error budget burns fast over both a short and a long window. In my simulation (200 requests per minute, assumed noise): on a sudden 40% outage the threshold rule paged at minute 0 and burn rate at minute 5; on a slow 3% burn for 4 hours the threshold rule paged at minute 24 only by chance, because random 1-minute spikes crossed 5%, and burn rate paged at 75; over 7 noise-only days the threshold rule paged 44 times, burn rate 2. Noisy pages train people to ignore them. Agent-specific signals to alert on: cost per minute, steps per run, tool error rate, approval queue depth, guardrail block rate, eval or replay drift (module 9), and refusal or escalation rate changes.
• Severity decides who wakes up and how fast. Example rules (mine, illustrative): data exposure or large money movement is SEV1; low availability, many tenants or any money moved is SEV2; wrong answers or mild degradation is SEV3. A cross-tenant leak is SEV1 regardless of size (module 7, Level 10 module 10 for notification duties). Say who is incident commander, who communicates, and who fixes.
• Mitigation is a trade. Simulated refund bug (5 bad actions per minute, detected after 8 minutes, all inputs assumed): waiting for a fix for 60 minutes means 340 bad actions; a rollback that takes 15 minutes means 115; a kill switch on the refund tool that takes 2 minutes means 50, at the cost of blocking legitimate refunds (720 if left on for an hour, 156 if lifted when the rollback lands at minute 15). A narrow, fast switch plus a rollback beat either alone.
• Runbooks are for the 3 a.m. brain. Each alert maps to a runbook with: owner, severity, first check, a concrete mitigation (a switch, a flag or a rollback command), rollback, who to escalate to, and the date it was last drilled. A runbook that says "investigate" has no mitigation. Practise with game days; stale runbooks fail when needed.
• The post-mortem is the improvement mechanism, and it is blameless. Required parts: summary, impact, timeline, root causes plural (contributing factors), how it was detected, what went well, and action items with an owner, a due date and a kind (prevent, detect or mitigate). Blameless means asking why the system allowed the mistake, not who made it. The usual agent-specific actions: add an eval for the failing case, add an alert for the symptom, add or tighten a kill switch, improve approval limits (Level 10 module 9), add the case to the replay set (module 9).
• Safe-change patterns reduce incidents: shadow mode (new version sees real inputs, outputs not served), canary with automatic rollback (module 8), eval gates, per-run budget caps (module 5), and human approval for irreversible actions.
• Honest limits: the simulations are small and the noise model, rates and severity rules are mine and assumed; no real incident process was run here.