CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 3 of 12 · 2 min read

The circuit breaker: stop hammering, fail fast

A switch that notices a dependency is failing and stops calling it.

Think of a household circuit breaker. When something downstream draws too much current, it trips and cuts power immediately. It does not let every appliance on the circuit keep drawing against a fault.

In plain words: a circuit breaker watches the recent failure rate, and once it crosses a threshold it stops sending calls to that dependency at all for a cooldown period. Callers then fail fast instead of waiting on calls that were going to fail anyway.

A worked example. With the breaker on its own during the outage, calls to the primary drop to 0.01 per request — essentially nothing but the occasional probe. Success is still only 68.3 percent, almost unchanged from having no protection at all, because the breaker's job is to stop hammering the dependency, not to find another way to succeed.

The key lines.

# from code/mini-reliability/reliability.py
class Breaker:
    def allow(self, now):
        if self.state == "closed": return True
        if self.state == "open" and now - self.opened >= self.cd: self.state, self.probe_out = "half", False
        if self.state == "half" and not self.probe_out: self.probe_out = True; return True
        return False

Read it as: while closed, every call goes through. Once open, calls are blocked until the cooldown passes. Then exactly one probe call decides whether to close again or stay open. That one probe is why a brief recovery can be detected. It does not flood the dependency with traffic to test it.

Circuit breaker, half-open

A circuit breaker blocks calls to a dependency once its recent failure rate crosses a threshold, re-testing it later with a single probe call; half-open is that probing state.

Example: 0.01 primary calls per request during the outage above — almost all of it blocked outright, the rest spent on probes.