Part 6 of 9 · 3 min read
Health checks and failure
How a balancer finds out that a server is dead, and what users see in the gap.
Think of a waiter who keeps seating guests at a table whose chair is broken. Someone has to tell the waiter, and the sooner, the fewer guests are hurt.
A load balancer must know which servers are healthy. The book names two kinds of check.
Health check
A health check is a test the balancer uses to decide whether a server should receive requests.
Example: every 5 seconds the balancer calls GET /health on each server. A 200 reply means healthy.
- Passive check. The balancer watches real traffic. If a server is unreachable, times out, or returns an error such as 503 that should not be retried, the balancer takes it out of the pool.
- Active check. The server offers a health endpoint, and the balancer calls it on a timer.
Passive checks cost nothing extra but find out only after a user has been hurt. Active checks find a problem before a user does, but add some background traffic.
A worked example of the gap. Three servers receive 600 requests per second in round robin, so each gets 200. Server 3 dies. The balancer sends an active check every 5 seconds, and declares a server dead after 3 failed checks in a row. The numbers are made up.
- Time until the balancer notices: up to 3 × 5 = 15 seconds.
- During that time one request in three goes to server 3.
- Failed requests: 200 per second × 15 seconds = 3,000.
How to shrink this loss:
- Check more often. Every 1 second with 3 failures needs only 3 seconds. Loss: 200 × 3 = 600. The cost is five times more check traffic.
- Add passive checks. A request that times out or fails marks the server at once.
- Retry on another server. The balancer re-sends a failed idempotent request (page 2) to a second server. The user sees only a slight delay.
A good health endpoint tells the truth about the thing that matters. If it only returns 200 while the process is running, a server that cannot reach its database still looks healthy. A better one checks the few things the server needs. But it must not check so much that one broken dependency removes every server at once. Level 8 returns to this.
The book also notes the other side of the story. A balancer can add servers to the pool or remove them at any time, which is how cloud systems add and drop servers as load changes (autoscaling). This needs a way for the balancer to learn which servers exist. The book lists a static file, DNS, or a data store, each with more flexibility and more complexity (Level 7 covers service discovery).
Read it as
Detection time × share of traffic on the dead server × request rate = requests lost. Shorter detection and retries reduce all three.
Check yourself:
- What is the difference between a passive and an active health check?
- In the worked example, what happens to the loss if the check runs every 2 seconds with 3 failures?