CSE Roadmap

Common tier · about 4 hours · 6 parts

Part 5 of 6 · 4 min read

Coordination services in practice

ZooKeeper and etcd keep small, critical facts that many other systems depend on.

Coordination services in practice

Think of a notice board at the front of a large office. It is small, everybody trusts it, and it is guarded very carefully. Nobody stores their whole filing cabinet on it.

The DDIA book describes ZooKeeper and etcd as small databases that hold data fitting in memory (written to disk for durability) and replicate it with a consensus algorithm. App developers rarely use them directly. HBase, Hadoop YARN, OpenStack Nova and Kafka use ZooKeeper in the background.

Features that make them useful:

  • Linearizable atomic operations, such as compare-and-set, which allow a lock. If several nodes try, only one succeeds. This is the only feature that truly needs consensus.
  • Total ordering of operations, which gives fencing tokens (module 4). ZooKeeper's transaction id can serve as one.
  • Failure detection: clients keep a session with heartbeats. If the heartbeats stop for longer than the session timeout, ZooKeeper declares the session dead, and locks held by it can be released automatically (ephemeral nodes).
  • Change notifications: clients can watch a key and learn when another client joins or fails, instead of polling.

Uses named in the book:

  • Choosing a leader or primary among instances of a service, and taking over on failure.
  • Assigning partitions to nodes, and moving them when nodes join or leave.
  • Service discovery (which address reaches a service). The book points out this does not really need consensus: DNS gives stale answers and that is usually fine. But if the consensus system already knows the leader, it can help others find it. Some systems offer read-only replicas that follow the decision log without voting, serving reads that need not be linearizable.
  • Membership: knowing which nodes are alive.

Another point of the book: they run on a fixed small number of nodes, usually three or five, and serve a large number of clients. This "outsources" consensus, ordering and failure detection to one service. The data is slow-changing, like "the node at 10.1.1.23 leads partition 7", changing over minutes or hours. It is not for state that changes thousands of times per second.

A worked example. A job system has 40 workers and 8 partitions of work.

Step Mechanism
A worker starts It creates an ephemeral node under /workers
Work is assigned A leader (elected by lock) writes "partition 3 → worker 12"
Worker 12 dies Its session times out, the ephemeral node disappears, watchers are told
Rebalancing The leader reassigns partition 3 to worker 27, with a new fencing token

Only the coordination service runs consensus (on 3 or 5 nodes). The 40 workers do not vote.

Common mistake

Writing a home-made consensus protocol or using a coordination service as a general database. Use a proven one, and store only small, slow-changing facts in it.

Interview angle

Answer in your head first, then open it.

What is consensus and why is it hard?

Agreement among nodes despite crashes and delays. Name the four properties.

How does Raft keep committed data safe?

Commit on a majority, and vote only for candidates with up-to-date logs, so every winner has all committed entries.

Why an odd number of nodes?

An even number tolerates no more failures than the odd number below it.

What do ZooKeeper and etcd do for you?

Locks, leader election, failure detection and configuration. Mention fencing tokens.

When not to use consensus?

When stale answers are fine (DNS-like discovery) or when speed matters more than strict order.

Check yourself:

  • Which feature of ZooKeeper needs consensus?
  • Why should application state that changes thousands of times a second not live in ZooKeeper?

Words you will meet

Word Plain meaning
Consensus Nodes agreeing on one proposed value
Propose, decide Offer a value, then settle on one
Uniform agreement No two nodes decide differently
Integrity No node decides twice
Validity The decision was proposed by someone
Termination Every live node eventually decides
FLP result A proof that consensus can fail in a model without clocks or timeouts
Total order broadcast All nodes get all messages in one order
Epoch or term A number that identifies one leader's period
Paxos, Raft, Zab, VSR Well-known consensus algorithms
Log An ordered list of operations
Committed entry A log entry stored on a majority
AppendEntries The Raft message that copies log entries and acts as a heartbeat
Quorum A majority needed for a decision
Ephemeral node A ZooKeeper entry that vanishes when its session dies
Coordination service A small consensus-backed store for locks and configuration