Part 4 of 8 · 1 min read
Prevention
Cut every link in the chain.
UDS's advice is that the best way to deal with a cascade is not to have one. Each earlier page cuts one link.
| Link in the chain | Defence | Page |
|---|---|---|
| A call that never returns | Timeout | 2 |
| Clients retrying at once | Backoff with jitter | 2 |
| Retries at several levels | Retry at one level only | 2 |
| Waiting on a broken dependency | Circuit breaker | 3 |
| One path using all the resources | Bulkhead | 3 |
| More work than capacity | Load shedding, rate limiting, queues | 4 |
| Losing the core when extras fail | Graceful degradation | 4 |
| Load moving onto the survivors | Spare capacity (headroom), health checks | 7, 8 |
Two more points come from the books.
Headroom. The five-server example showed that spare capacity decides whether a failure stays local. UDS says to prepare beforehand for load you do not expect. A rule of thumb is to size so that the loss of one failure unit (a server, a zone) still leaves the others below their limit. (How much headroom to keep is a business question. The books do not give a number.)
Risk scoring. Not every cascade deserves the same effort. Score likelihood times impact (page 1) and spend most on the chains that touch critical paths.
Common mistake
Adding tools one at a time and testing each alone. A circuit breaker tested alone looks fine. The cascade comes from how retries, pools, load balancers and breakers act together. Test the combination with a failure drill (page 6).
Check yourself: 1. Which two tools cut the retry link? 2. What does headroom protect against? 3. Why must tools be tested together?