CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 3 of 12 · 2 min read

Mitigate first, diagnose second: the kill-switch ladder

Stopping the damage beats understanding it, and a narrow switch costs less than a rollback alone.

Think of pulling a fire alarm versus first trying to figure out exactly which outlet started the fire. The alarm stops the fire from spreading right now; the investigation can happen once everyone is safely out.

In plain words: when an agent is actively doing harm, flipping a narrow kill switch to stop it is faster and cheaper than waiting for a proper fix or even a full rollback, even though the switch itself blocks some legitimate work while it's on.

A worked example. A refund bug causes 5 bad refunds per minute, detected after 8 minutes. Waiting for a proper fix that takes 60 minutes to ship results in 340 bad actions by the time it lands. Rolling back the release in 15 minutes cuts that to 115. Flipping a kill switch on just the refund tool, which takes 2 minutes to apply, cuts it to 50 — at the cost of 720 legitimate refunds blocked if left on for a full hour, or just 156 if lifted the moment the 15-minute rollback lands.

The key lines.

# from code/mini-incident/mitigation.py
def harm(rate_per_min, detect_min, mitigations):
    for name, apply_min, collateral, minutes_on in mitigations:
        yield {"name": name, "minutes_to_stop": detect_min + apply_min,
               "bad_actions": rate_per_min * (detect_min + apply_min),
               "legit_blocked": collateral * minutes_on}

Read it as: bad actions keep happening until detection plus however long the chosen fix takes to apply. A 2-minute switch stops the clock almost immediately. A 60-minute proper fix lets the bug keep running the whole time it is being written. Combining a fast switch with a rollback, then lifting the switch once the rollback lands, gets both numbers down at once.

Kill switch

A control that disables one capability immediately, with no deploy required — scoped to a tool, a tenant or the whole system, and fail-closed so a failure in the switch itself still blocks the capability.

Example: the 50-bad-actions outcome above, the kill switch's own result, well below either the rollback's 115 or the full-fix wait's 340.