CSE Roadmap

Common tier · about 3 hours · 6 parts

Part 2 of 6 · 2 min read

Detecting failures

Guess whether a peer is down, using pings, heartbeats and timeouts.

Think of a night shift where each guard must call the control room every ten minutes. If a call is missed, the control room sends someone to check. A missed call does not prove the guard is hurt: the phone line may be down. It only says something is probably wrong.

The book explains the same limit. When a client sends a request and nothing comes back, the server may be slow, crashed, or the message may have been lost or delayed. The best a client can do is make an educated guess with a timeout. The trouble is choosing the length: too short, and a reachable server is wrongly treated as dead; too long, and the client blocks waiting for a server that is gone. The book states plainly that a perfect failure detector cannot be built.

Failure detector

A failure detector is a part of a system that guesses which other processes are alive, usually from messages that arrive in time and messages that do not.

Example: a load balancer marks a server dead if it misses three health checks in a row, and marks it alive again when a check succeeds.

Two common tools:

  • A ping is a periodic request to another process, with an answer expected within a set time. If none comes, a timeout marks the target dead. The pinger keeps sending, so if the target returns, it answers and is marked available again.
  • A heartbeat is a message a process sends on a schedule to say it is still running. If the receiver gets none within a set time, it marks the sender dead. When heartbeats resume, it is marked available.

The difference is who starts: the checker asks in a ping, and the checked process tells in a heartbeat.

When to use them, per the book: pings and heartbeats suit processes that talk to each other often, where something must happen as soon as one disappears. Otherwise, discovering a failure only at the moment of sending a request is good enough.

A worked example. A leader sends a heartbeat every 1 second. Followers wait up to 5 seconds before they suspect it.

Setting Effect
Wait 2 s A single delayed heartbeat (a pause of 2 s) triggers a false alarm
Wait 5 s Tolerates short stalls. A real crash takes up to 5 s to notice
Wait 30 s Almost no false alarms. A crash leaves the system without a leader for up to 30 s

False alarms in a leader system are costly, because they start an election, which briefly stops work (see section 3). Slow detection is costly, because users wait. The right value comes from measuring the network, as module 4 said.

Check yourself:

  • What is the difference between a ping and a heartbeat?
  • Why can a failure detector never be perfect?