CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 6 of 7 · 3 min read

Trade-offs

Small and safe, not big and fast.

Scale, reliability and trade-offs

Sizing.

  • Data: 500,000 entries × 1 KB = 500 MB. It fits in memory on each replica.
  • Writes: a few hundred a second. Each is one consensus round, taking milliseconds in one region.
  • Reads and watches: tens of thousands a second, spread over all replicas. Add observer replicas that receive the log but do not vote, so read capacity grows without slowing writes.
  • Sessions: 100,000 sessions with heartbeats every 5 seconds is 20,000 a second.
  • A very large user base is served by several separate groups (one per region or per application area). A single group is not sharded in the usual sense, since total order is its purpose.

Replica count. 3 replicas are common, 5 give more safety (two failures, or one failure plus one in maintenance). Beyond 5 or 7, each write waits for more machines and gains little. Replicas should sit in different failure zones. Spreading across far-apart regions slows every write by the network round trip.

Failure behaviour.

  • A replica fails: the group continues if a majority remains. Clients move to another replica and keep their sessions.
  • The leader fails: an election takes seconds. Writes pause until it ends.
  • A majority is lost: the service stops accepting writes, since no safe decision can be made. It is better to stop than to let two sides decide differently.
  • Network split: only the side with a majority continues.

Session timeout trade-off. A short timeout detects crashes quickly but may mistake a slow client for a dead one (a false failover). A long timeout is calmer but slower to react. Typical values are seconds to tens of seconds.

Choice Gain Cost
Consensus group of 3 or 5 Safe, simple Limited write rate
Ephemeral entries Automatic cleanup A slow client can lose its entry
Leases No stuck locks Need fencing for safety
Fencing tokens Correct under pauses The resource must check
Watches Fast reaction, less polling Many watchers can cause bursts of traffic
Observers More read capacity Slightly stale reads
Short session timeout Fast failover More false alarms

Interview angle

Answer in your head first, then open it.

How do you implement a distributed lock?

A create-if-absent ephemeral entry in a consensus store, with a lease and a fencing token.

What can go wrong with a lease?

A paused holder believes it still holds the lock. The resource must reject old fencing tokens.

How does leader election work?

Candidates race to create an ephemeral entry. Others watch it.

Why not a normal database?

A coordination service gives consensus, sessions, ephemeral entries and watches together.

What if the service loses a majority?

It stops writes. Clients must treat their locks as lost.

One follow-up to try. "Why not use Redis for locks?" A good reply: a single Redis node is a single point of failure, and a replicated Redis can lose an acknowledged write when a failover happens, so two clients might hold the same lock. If the lock only protects efficiency (a duplicate job wastes a little time), that may be fine. If it protects correctness, use a consensus-based store and fencing tokens at the resource.

Check yourself: 1. Why do many separate groups serve large user bases? 2. What happens when a majority is lost? 3. When is a weaker lock acceptable?

Words you will meet

Word Meaning
Coordination service A small consensus store for locks, leaders and config
Quorum A majority of the replicas
Session A client's link to the service, kept alive by heartbeats
Ephemeral entry An entry that vanishes with its session
Watch A request to be told when an entry changes
Lease A right that expires unless renewed
Fencing token A rising number that lets a resource reject old holders
CAS Compare-and-set: change only if the version matches