CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 2 of 7 · 2 min read

What lag is

The delay between a write on the leader and its arrival on a follower.

Think of a news agency that sends reports to local newspapers by post. A report written in the capital on Monday is printed in a small town on Wednesday. For two days, the capital and the town tell different stories. If the postal service strikes, the town may be a week behind.

The book "Designing Data-Intensive Applications" explains that replication is useful for failure tolerance, for scale and for latency. The read-scaling architecture adds many followers and spreads read-only queries across them. It only realistically works with asynchronous replication. With synchronous copies to all followers, one node failure or network break would stop all writes, and the more nodes there are, the likelier one is down.

The price is that a follower may show old data.

Replication lag

Replication lag is the delay between a write being made on the leader and the same write appearing on a follower.

Example: a user posts a comment at 10:00:00.000. The follower applies it at 10:00:00.350. The lag was 350 milliseconds.

Eventual consistency

Eventual consistency means that if writes stop, all replicas will in time hold the same data. It says nothing about how long that takes.

Example: after the last write, a follower that was 40 seconds behind catches up and now matches the leader.

The word "eventually" is deliberately vague. The book says there is no limit on how far a replica can fall behind. In normal running the lag is a fraction of a second and unnoticed. Near capacity, or with network faults, it can reach seconds or minutes. The book also points out that eventual consistency is not only a NoSQL trait: followers of an asynchronously replicated relational database behave the same way.

A worked example. A leader applies 3,000 writes per second. A follower can apply 3,500 per second. It keeps up, with a spare 500 per second. A burst of 6,000 writes per second lasts 10 seconds. The follower falls behind by (6,000 − 3,500) × 10 = 25,000 writes. After the burst, it clears the backlog at 500 per second: 25,000 ÷ 500 = 50 seconds. So a 10-second burst gives a lag that peaks at about 25,000 ÷ 3,500 ≈ 7 seconds, and the follower is behind for a minute in total.

The result is that if the same query runs on the leader and a follower at the same time, the answers may differ. Three kinds of problem follow from this.

Anomaly What the user sees Guarantee that prevents it
Lost own write "I saved it, and it is gone" Read-your-writes
Time goes backward A comment appears, then disappears Monotonic reads
Answer before question A reply is seen before the message it answers Consistent prefix reads

Check yourself:

  • A follower applies 2,000 writes per second. A burst of 5,000 writes per second lasts 4 seconds. How large is the backlog at the end of the burst?
  • Why is "eventually" a loose promise?