CSE Roadmap

Core tier · about 3 hours · 9 parts

Part 6 of 9 · 3 min read

Health checks and failure

How a balancer finds out that a server is dead, and what users see in the gap.

Think of a waiter who keeps seating guests at a table whose chair is broken. Someone has to tell the waiter, and the sooner, the fewer guests are hurt.

A load balancer must know which servers are healthy. The book names two kinds of check.

Health check

A health check is a test the balancer uses to decide whether a server should receive requests.

Example: every 5 seconds the balancer calls GET /health on each server. A 200 reply means healthy.

  • Passive check. The balancer watches real traffic. If a server is unreachable, times out, or returns an error such as 503 that should not be retried, the balancer takes it out of the pool.
  • Active check. The server offers a health endpoint, and the balancer calls it on a timer.

Passive checks cost nothing extra but find out only after a user has been hurt. Active checks find a problem before a user does, but add some background traffic.

A worked example of the gap. Three servers receive 600 requests per second in round robin, so each gets 200. Server 3 dies. The balancer sends an active check every 5 seconds, and declares a server dead after 3 failed checks in a row. The numbers are made up.

  • Time until the balancer notices: up to 3 × 5 = 15 seconds.
  • During that time one request in three goes to server 3.
  • Failed requests: 200 per second × 15 seconds = 3,000.

How to shrink this loss:

  1. Check more often. Every 1 second with 3 failures needs only 3 seconds. Loss: 200 × 3 = 600. The cost is five times more check traffic.
  2. Add passive checks. A request that times out or fails marks the server at once.
  3. Retry on another server. The balancer re-sends a failed idempotent request (page 2) to a second server. The user sees only a slight delay.

A good health endpoint tells the truth about the thing that matters. If it only returns 200 while the process is running, a server that cannot reach its database still looks healthy. A better one checks the few things the server needs. But it must not check so much that one broken dependency removes every server at once. Level 8 returns to this.

The book also notes the other side of the story. A balancer can add servers to the pool or remove them at any time, which is how cloud systems add and drop servers as load changes (autoscaling). This needs a way for the balancer to learn which servers exist. The book lists a static file, DNS, or a data store, each with more flexibility and more complexity (Level 7 covers service discovery).

Read it as

Detection time × share of traffic on the dead server × request rate = requests lost. Shorter detection and retries reduce all three.

Check yourself:

  • What is the difference between a passive and an active health check?
  • In the worked example, what happens to the loss if the check runs every 2 seconds with 3 failures?