CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 5 of 8 · 2 min read

Watchdogs

A process that checks itself and restarts if it is sick.

UDS builds on a design idea: since any process can crash at any time, the service must be able to cope. The conditions it lists:

  • other identical processes can handle incoming requests;
  • requests are stateless and any process can serve them;
  • any non-volatile state is stored in a separate, dedicated data store, so a crash loses nothing;
  • shared resources are leased, so when a process crashes the leases expire and others can use the resources;
  • the service always runs slightly over-scaled, to withstand the odd process failure.

When these hold, you do not need complex recovery logic for a process in a strange degraded state. You can simply let it crash. A rare, short-lived failure can be hard to diagnose. Crashing and restarting gives operators breathing room until the cause is found, which is a form of self-healing.

The book's example is a memory leak. Left alone, memory shrinks, the process swaps to disk, and the whole fleet slows. Would you rather have the process notice and restart itself, or debug the root cause at 3 a.m.?

The tool is a watchdog: a separate background thread that wakes up periodically and checks the process's health, for example the free physical memory. When a metric crosses its threshold, the watchdog regards the process as degraded and restarts it on purpose.

Watchdog

A watchdog is a part of a process (or a monitor beside it) that checks its health on a timer and restarts it when a threshold is crossed.

Example: a watchdog checks free memory every 10 seconds and restarts the process if less than 5 percent is free for 3 checks in a row.

UDS warns that a watchdog needs good tests and monitoring, since a fault in it can make processes restart again and again.

Restart loops are a real danger. Practice (not from the books used here) adds guards:

  • Require several bad readings in a row before acting, to avoid reacting to a blip.
  • Limit the restarts per hour, and escalate to a human past the limit.
  • Wait longer between restarts (backoff, page 2), and restart instances one at a time, not all at once. If all restart at once the service is down.
  • Record each restart as an event, with the reason, so the pattern is visible.

A worked example (made up). A leak costs 100 MB per hour. A process has 8 GB, and the watchdog restarts it when under 1 GB remains. That happens after 70 hours. With 10 servers the restarts are spread by start time, and the fleet loses at most one server for about 30 seconds each three days. Without the watchdog, all 10 would die within the same hour, since they started together after the last release. The staggered restart prevents a cascade (page 5).

Check yourself: 1. Why does "let it crash" need stateless requests? 2. What can go wrong with a watchdog? 3. Why should instances be restarted one at a time?