Part 4 of 8 · 2 min read
The safety rule
When everyone looks sick, the checker is the likely culprit.
UDS marks the dependency check with "here be dragons". If the dependency is briefly unreachable, or the check has a bug, it is possible that every process behind the balancer fails the check. A naive balancer would remove all instances and bring the entire service down.
The book's fix: a smart load balancer notices that a large fraction of the instances is reported unhealthy and decides that the health check is no longer reliable. Instead of removing more processes, it ignores the health checks and sends requests to any process in the pool. This is sometimes called "failing open".
A worked example (made up to show the idea). A rule: if more than 50 percent of instances are unhealthy, stop removing and ignore the checks. With 8 servers, the first 4 are removed (50 percent). When the fifth fails the check, the balancer sees that 5 of 8 are unhealthy and goes into fail-open mode, sending traffic to all 8 again. Most requests still fail on the dependency, but those that do not need it succeed, and the system recovers as soon as the dependency does. (The 50 percent threshold is an example. The book does not give a number.)
Common mistake
Making the health check depend on every downstream service. A check that goes red when an optional dependency is down turns a partial failure into a total one. Check the dependencies that the core function cannot work without, and treat the rest as degradation (page 4).
Related rule from practice (not in the books used): separate liveness (restart me if I fail) from readiness (send me traffic only when I pass). A server that is starting, or loading a cache, is alive but not yet ready. Container platforms such as Kubernetes use both ideas.
Check yourself: 1. What does a smart balancer do when most servers fail the check? 2. Why is "fail open" safer than removing everyone? 3. Which dependencies belong in a health check?