Part 3 of 12 · 2 min read
Mitigate first, diagnose second: the kill-switch ladder
Stopping the damage beats understanding it, and a narrow switch costs less than a rollback alone.
Think of pulling a fire alarm versus first trying to figure out exactly which outlet started the fire. The alarm stops the fire from spreading right now; the investigation can happen once everyone is safely out.
In plain words: when an agent is actively doing harm, flipping a narrow kill switch to stop it is faster and cheaper than waiting for a proper fix or even a full rollback, even though the switch itself blocks some legitimate work while it's on.
A worked example. A refund bug causes 5 bad refunds per minute, detected after 8 minutes. Waiting for a proper fix that takes 60 minutes to ship results in 340 bad actions by the time it lands. Rolling back the release in 15 minutes cuts that to 115. Flipping a kill switch on just the refund tool, which takes 2 minutes to apply, cuts it to 50 — at the cost of 720 legitimate refunds blocked if left on for a full hour, or just 156 if lifted the moment the 15-minute rollback lands.
The key lines.
# from code/mini-incident/mitigation.py
def harm(rate_per_min, detect_min, mitigations):
for name, apply_min, collateral, minutes_on in mitigations:
yield {"name": name, "minutes_to_stop": detect_min + apply_min,
"bad_actions": rate_per_min * (detect_min + apply_min),
"legit_blocked": collateral * minutes_on}
Read it as: bad actions keep happening until detection plus however long the chosen fix takes to apply. A 2-minute switch stops the clock almost immediately. A 60-minute proper fix lets the bug keep running the whole time it is being written. Combining a fast switch with a rollback, then lifting the switch once the rollback lands, gets both numbers down at once.
Kill switch
A control that disables one capability immediately, with no deploy required — scoped to a tool, a tenant or the whole system, and fail-closed so a failure in the switch itself still blocks the capability.
Example: the 50-bad-actions outcome above, the kill switch's own result, well below either the rollback's 115 or the full-fix wait's 340.