CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 2 of 8 · 2 min read

Why health checks

Stop sending traffic to a server that cannot serve it.

Page 4 showed how a server can reject requests after they arrive. That cost something: the request already reached it. UDS asks for better: a way to keep traffic from reaching a degraded server at all.

If the server is behind a load balancer, it can tell the balancer that it is overloaded or broken. It exposes a health endpoint: an address the balancer asks every few seconds. The endpoint answers 200 (OK) when the process can take requests, and an error code when it cannot. If the endpoint returns an error, or does not answer in time, the balancer takes the process out of its pool.

UDS gives the reason it matters for availability. With 10 servers and one unresponsive, 10 percent of the requests will fail, and the availability of the service drops to 90 percent. A health check lets the balancer stop that.

Health check

A health check is a periodic test that tells whether a process can do its work. A load balancer or orchestrator uses the result to include or remove it.

Example: every 5 seconds the balancer calls GET /health. Three failures in a row remove the server from rotation.

Behind this is the problem of failure detection (UDS chapter 7). A remote process that does not answer could be dead, slow, or cut off by the network, and from outside you cannot tell which (page 1). DDIA's chapter 8 says the same: a timeout is the only general way to detect a fault, and there is no perfect value. Too short, and healthy nodes are declared dead. Too long, and users wait on a dead node. A node declared dead falsely has its work moved to others, which adds load to them.

Check yourself: 1. Why does a health check beat load shedding for keeping traffic away from a bad server? 2. If 1 server in 20 is unresponsive and not removed, what share of requests fail? 3. Why is the timeout for a health check a trade-off?