Part 5 of 6 · 3 min read
The three together
Each covers a gap the others leave.
Putting the three together
| Tool | Protects against | Does not help with |
|---|---|---|
| Timeout | One call hanging for ever | A dependency that stays broken, because each call still waits the full timeout |
| Breaker | A dependency that is down for a longer time | Resources already used up, and the first failures before it trips |
| Bulkhead | One failing path using all shared resources | Failures of the whole shared service itself |
BM's advice is to put timeouts on all out-of-process calls with a default value, log when they fire, and adjust. Use separate pools for each downstream service at least. Consider mandating circuit breakers for all synchronous downstream calls, since cascading failures are so costly. You need not write them yourself. BM names Netflix's Hystrix for the JVM, Polly for .NET and a mixin for Ruby. (Hystrix is now in maintenance mode; the idea has moved into other libraries and into service meshes, which is not covered in the books used here.)
BM also notes that a Hystrix-style bulkhead can reject requests in some conditions, so resources do not get further saturated. That is load shedding, the subject of page 4. Sometimes refusing a request is the best way to keep an important system from becoming a bottleneck for many callers.
BM links all this to isolation. When one service needs another to be running, the health of the second flows into the first. Integration styles that let a downstream service be offline (queues, Level 6) leave callers less exposed to outages, planned or not. Isolation also means less coordination between teams.
Interview angle
Answer in your head first, then open it.
What is a circuit breaker?
A guard that counts failures to a dependency, opens after a threshold so calls fail at once, and tests the dependency again after a pause (half-open).
What do you do while it is open?
Degrade gracefully: default value, hide the feature, queue the work or return a clear error.
What is a bulkhead?
Separate pools or partitions, so that one failing path cannot use all the resources.
Why is a slow service worse than a dead one?
Callers wait, hold threads and connections, and the slowness spreads. A dead service fails fast.
Which of timeouts, breakers and bulkheads matters most?
BM argues for bulkheads, since they stop resources running short in the first place, but you want all three.
Check yourself: 1. What gap does a breaker leave that a bulkhead fills? 2. Why is a timeout alone not enough for a dependency that is down for ten minutes? 3. Which of the three would you add first if you could add only one?
Words you will meet
| Word | Meaning |
|---|---|
| Fail fast | Report an error at once instead of waiting |
| Circuit breaker | A guard that stops calls to a failing dependency |
| Closed / open / half-open | The three states of a breaker |
| Threshold | The number of failures that trips the breaker |
| Graceful degradation | Keeping the core working with fewer features |
| Bulkhead | A split of a shared resource into separate parts |
| Connection pool | A fixed set of reusable connections |
| Virtual partition | A random subset of instances assigned to a client |
| Load shedding | Rejecting some requests to protect the service |