Part 3 of 6 · 3 min read
The circuit breaker
Stop calling something that is failing.
A house has circuit breakers that cut the power when a spike could harm the appliances. BM and UDS borrow the idea. After a certain number of calls to a downstream service fail, the breaker "blows". All further calls fail at once, without touching the network. UDS puts it neatly: the fastest network call is the one you do not make.
Retries and breakers answer different expectations. A retry helps when the next call is expected to succeed. A breaker helps when the next call is expected to fail.
Circuit breaker
A circuit breaker is a guard around a remote call. After too many failures it stops sending calls and fails them at once. After a pause it tests whether the service has recovered.
Example: after 5 timeouts in 10 seconds, the checkout stops calling the loyalty service for 30 seconds.
UDS describes a state machine with three states.
| State | What happens | Moves on when |
|---|---|---|
| Closed | Calls pass through. The breaker counts failures (errors and timeouts) | The count passes a threshold within a time window: go to open |
| Open | Calls are not attempted and fail immediately | After a set time: go to half-open |
| Half-open | The next call is let through as a test | It succeeds: go to closed. It fails: back to open |
BM uses the words "blown" and "reset". He defines a failed call for HTTP as a timeout or a 5xx status code. In his version, the breaker sends a few calls through after a while, and if it gets enough healthy answers it resets.
A worked example (made up). Settings: trip at 5 failures in 10 seconds, stay open 30 seconds.
| Time | Event | State |
|---|---|---|
| 0 s | Loyalty service starts timing out | Closed |
| 6 s | Fifth timeout within 10 s | Open |
| 6 to 36 s | Every call fails in under 1 ms | Open |
| 36 s | One test call is let through, times out | Open again |
| 66 s | Test call succeeds | Closed |
Without the breaker, with a 2 second timeout and 100 calls per second, the caller would hold 200 threads busy waiting. With the breaker, it holds almost none during the open period.
Choosing the settings. How many failures are enough? How long to wait before testing? UDS and BM both say the answer depends on the case and on data about past failures. BM's advice: pick sensible defaults and use them everywhere, then change them for special cases. Do not trip too eagerly, and do not take too long to trip.
What to do while open. This is a business decision. UDS says that if the dependency is not critical, the service should degrade gracefully and not stop. Its examples are a plane that loses a non-critical system and still lands, and the Amazon front page that shows no recommendations when that service is down. BM lists options: queue the work and retry later if it is an async job, or in a synchronous chain fail fast, either by passing an error up or by removing a feature.
BM adds a use for a hand-operated breaker. Before taking a service down for maintenance, blow the breakers of its callers. They fail fast while it is offline, and you reset them when it returns.
Common mistake
Putting a breaker in but giving no thought to what happens when it opens. A breaker that turns a slow failure into a fast failure is only half the work. The caller needs a plan: a default value, a hidden feature or a clear error.
Check yourself: 1. Which state lets exactly one test call through? 2. In the table, what happens if the 66 s test call fails? 3. Why does a recommendations service make a good case for graceful degradation?