Part 5 of 12 · 2 min read
Escalating to a different person beats paging the same one again, and a breaker buys time
One person's unavailability is correlated across a whole incident, so moving to a different tier after a timeout reaches a human far more often than re-paging the same one, and a circuit breaker halts the agent while that paging happens.
Think of a fire alarm that calls the next shift if the first desk does not answer, rather than redialling the same desk every few minutes.
Because one person's unavailability is correlated across an entire incident, paging a different tier after a timeout reaches a human far more reliably than re-paging the same person. A circuit breaker halts the agent automatically while that paging happens.
A worked example. At 70 percent availability, re-paging the same on-call person every 15 minutes never reached a human in 30.0 percent of incidents. Escalating through on-call engineer, then manager, then security lead cut that to 2.5 percent, with 95 percent of incidents acknowledged within 39 minutes. In a simulated attack of 6 attempts a minute, a breaker set to open at 3 suspicious events in 60 seconds cut harm from 68.5 units with no breaker to 2.3, at the cost of about 3 false trips over 8 hours of ordinary traffic.
The key lines. From code/mini-accountability/accountability.py:
# from code/mini-accountability/accountability.py
def record(self, ts, suspicious):
if self.open: return False
if suspicious: self.events.append(ts)
self.events = [t for t in self.events if ts - t <= self.window]
if len(self.events) >= self.limit: self.open = True
return not self.open
Read it as: the breaker only counts events inside a sliding window. Three suspicious actions within 60 seconds open it, but the same 3 events spread across an hour would not. The window, not just the count, decides whether it trips.
Circuit breaker, kill switch
- Circuit breaker: an automatic halt when a threshold of bad events is reached.
- Kill switch: a manual or automatic control that stops the agent or its tools immediately.
Example: with a limit of 3 suspicious events in 60 seconds, a simulated attack that would otherwise cause 68.5 units of harm was cut to about 2.3 before the breaker opened and a named human had to reset it.