Part 3 of 8 · 2 min read
Kinds of checks
Is it running, is it fit, can it reach what it needs.
UDS describes three levels, from simple to hard.
| Kind | What it checks | Example | Danger |
|---|---|---|---|
| Liveness | Does the process answer a basic request with 200? | GET /health returns OK |
A process can answer but be unable to do real work |
| Local health | Is a local resource near its limit? The process compares memory, CPU or disk with thresholds | Report unhealthy if free memory is below 200 MB or disk is over 95 percent full | Wrong thresholds cause flapping or miss real problems |
| Dependency health | Are the remote dependencies working? The process measures response time, timeouts and errors of calls to them | Report unhealthy if more than 50 percent of database calls time out | If the dependency is down for everyone, every server reports unhealthy |
A liveness check proves the process is alive. A local check proves it is fit. A dependency check tries to prove it can do its job. The third is the hardest to get right.
Liveness, local and dependency health
Liveness asks "is it running?". Local health asks "are my own resources fine?". Dependency health asks "can I reach what I need?".
Example: a web server answers liveness, reports 80 percent memory use in the local check, and fails the dependency check when its database is down.
A worked example (made up). A service has 8 servers behind a balancer. Each reports unhealthy when the database fails more than half its calls. The database has a 2 minute outage. All 8 servers fail the dependency check within 15 seconds. A naive balancer takes all 8 out of rotation. The service now returns errors to everyone, although some requests (cached pages, static files, a "try again later" page) could have been served. The outage of one dependency has become a total outage.
Check yourself: 1. Which kind of check can catch a full disk? 2. Why can a liveness check pass while the service is useless? 3. In the example, what did the dependency check make worse?