CSE Roadmap

Core tier · about 3 hours · 6 parts

Part 3 of 6 · 3 min read

The circuit breaker

Stop calling something that is failing.

A house has circuit breakers that cut the power when a spike could harm the appliances. BM and UDS borrow the idea. After a certain number of calls to a downstream service fail, the breaker "blows". All further calls fail at once, without touching the network. UDS puts it neatly: the fastest network call is the one you do not make.

Retries and breakers answer different expectations. A retry helps when the next call is expected to succeed. A breaker helps when the next call is expected to fail.

Circuit breaker

A circuit breaker is a guard around a remote call. After too many failures it stops sending calls and fails them at once. After a pause it tests whether the service has recovered.

Example: after 5 timeouts in 10 seconds, the checkout stops calling the loyalty service for 30 seconds.

UDS describes a state machine with three states.

State What happens Moves on when
Closed Calls pass through. The breaker counts failures (errors and timeouts) The count passes a threshold within a time window: go to open
Open Calls are not attempted and fail immediately After a set time: go to half-open
Half-open The next call is let through as a test It succeeds: go to closed. It fails: back to open

BM uses the words "blown" and "reset". He defines a failed call for HTTP as a timeout or a 5xx status code. In his version, the breaker sends a few calls through after a while, and if it gets enough healthy answers it resets.

A worked example (made up). Settings: trip at 5 failures in 10 seconds, stay open 30 seconds.

Time Event State
0 s Loyalty service starts timing out Closed
6 s Fifth timeout within 10 s Open
6 to 36 s Every call fails in under 1 ms Open
36 s One test call is let through, times out Open again
66 s Test call succeeds Closed

Without the breaker, with a 2 second timeout and 100 calls per second, the caller would hold 200 threads busy waiting. With the breaker, it holds almost none during the open period.

Choosing the settings. How many failures are enough? How long to wait before testing? UDS and BM both say the answer depends on the case and on data about past failures. BM's advice: pick sensible defaults and use them everywhere, then change them for special cases. Do not trip too eagerly, and do not take too long to trip.

What to do while open. This is a business decision. UDS says that if the dependency is not critical, the service should degrade gracefully and not stop. Its examples are a plane that loses a non-critical system and still lands, and the Amazon front page that shows no recommendations when that service is down. BM lists options: queue the work and retry later if it is an async job, or in a synchronous chain fail fast, either by passing an error up or by removing a feature.

BM adds a use for a hand-operated breaker. Before taking a service down for maintenance, blow the breakers of its callers. They fail fast while it is offline, and you reset them when it returns.

Common mistake

Putting a breaker in but giving no thought to what happens when it opens. A breaker that turns a slow failure into a fast failure is only half the work. The caller needs a plan: a default value, a hidden feature or a clear error.

Check yourself: 1. Which state lets exactly one test call through? 2. In the table, what happens if the 66 s test call fails? 3. Why does a recommendations service make a good case for graceful degradation?