CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 5 of 7 · 2 min read

Recipes

Leader election, discovery, configuration.

Leader election. Each candidate tries to create the ephemeral entry /service/leader with its own address. One succeeds and is the leader. The others watch the entry. If the leader's session dies, its entry disappears, every follower is told, and they race to create it again. The new leader gets a higher fencing token, which it passes to the resources it uses, as in the scheduler (page 7).

A fairer version avoids a stampede: each candidate creates a numbered ephemeral entry in order (/election/n_0001, n_0002, ...). The lowest number is leader. Each other candidate watches only the entry just below its own. When the leader dies, only the next candidate wakes up.

Service discovery and membership. Each server creates an ephemeral entry such as /services/search/server-7 with its address. Clients list the children of /services/search and watch for changes. Servers appear and disappear on their own. In the nearby friends design (page 1), the hash ring of the pub/sub servers is kept here, and the WebSocket servers watch it.

Configuration. Keep settings under /config/.... Applications read them at start and watch for changes. Because all replicas see the same order, every client moves from one version to the next in the same order.

Work assignment. Partitions of a resource are assigned to nodes. Each node creates an ephemeral entry, and an assigner (itself elected a leader) reads the member list and writes the partition map. When a node vanishes, the assigner moves its partitions elsewhere.

Barriers and counters. CAS and ordering also support barriers ("wait until 5 members are present") and exact counters.

Good citizen rules for clients.

  • Do not store large data. Keep values to a few KB.
  • Do not poll in tight loops. Use watches.
  • Handle session expiry: a client that loses its session must assume it lost all its locks and entries and stop acting as leader at once.
  • Do not make every request depend on the service. Cache the answers, and degrade gently if it is unreachable. A base service that many depend on should not be on every request path.

A worked example (made up). A cluster of 5 search servers registers under /services/search. A client lists 5 children, caches the addresses, and watches. Server 3 crashes, and its session times out after 10 seconds. Its entry disappears. The client gets a notice and drops server 3 from its list within a moment. Total time to react: about 10 seconds, set by the session timeout.

Read it as

Sessions turn "this process died" into "this entry vanished". Watches turn that into "everyone who cares has been told".

Check yourself: 1. How does a follower learn that the leader died? 2. Why is the numbered election variant fairer? 3. What must a client do if its session expires?