CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 6 of 7 · 4 min read

Living with lag

A way to decide how much consistency an application needs, and what it costs.

Living with lag

Think of a shop that tells customers that stock levels on the website are updated every few minutes. Most customers do not mind. The same shop would not accept a delay of minutes in its till, where a wrong total costs money.

The book gives a test for eventually consistent systems: think about how the application behaves if the lag grows to several minutes or even hours. If the answer is "no problem", good. If users would have a bad experience, the design needs a stronger guarantee such as read-your-writes. It warns that pretending replication is synchronous when it is asynchronous is a recipe for problems.

An application can add guarantees on top of the database, such as reading some data from the leader. But that is complex and easy to get wrong. The book says it would be better if developers could trust the database to do the right thing. Transactions exist for this reason: they let the database give stronger guarantees so that the application is simpler. Many distributed databases gave up transactions, claiming they cost too much in performance and availability. The book says there is some truth in that, but it is overly simple. It promises a more nuanced view later.

A procedure to choose:

  1. List the reads of the application.
  2. For each, ask: what would a user see if the data were 1 minute old? 1 hour old?
  3. Mark reads where staleness hurts (own profile after an edit, an order just placed, a bank balance).
  4. Give those reads the cheapest guarantee that fixes the problem: leader reads, session stickiness, or position tracking.
  5. Leave the rest on any follower.

A table of examples:

Read If stale Guarantee needed
Home feed of other people's posts Slight delay, unnoticed None
Own post after publishing Looks lost Read-your-writes
Comments under a post, refreshed A comment flickers Monotonic reads
Chat messages Reply before question Consistent prefix
Account balance before payment Wrong decision Read from the leader
Product stock count in a list Slightly wrong, fine None
Stock count at checkout Oversell Read from the leader, or a transaction

A worked example of the cost-benefit. A site sends 90 percent of reads to followers. Adding read-your-writes for own-profile reads raises leader reads by 3 percent of traffic. At 40,000 reads per second, that is 1,200 more leader reads per second. If the leader has 6,000 spare operations per second, the extra load is acceptable and the guarantee costs 20 percent of the spare capacity.

Interview angle

Answer in your head first, then open it.

What is replication lag, and what problems does it cause?

The delay before a follower shows a write. It causes lost own writes, time running backward and effects seen before causes.

How would you give a user their own writes?

Read recently changed data from the leader, or track the position of the user's last write and serve from a follower that has reached it. Mention multiple devices.

How do you stop a refresh from showing older data?

Route each user to the same follower, for example by hashing the user id.

Is eventual consistency acceptable here?

Walk through each read: what happens if it is a minute old. Give the strong guarantee only to reads that need it.

How could the lag grow large?

Bursts of writes, a follower recovering, network faults, a very slow query on the follower. Monitor lag and take lagging followers out of rotation.

Check yourself:

  • Name two reads in a shop that can tolerate lag and two that cannot.
  • Why is it better to rely on database guarantees than on special code in each application?

Words you will meet

Word Plain meaning
Replication lag The delay between a write on the leader and on a follower
Eventual consistency Replicas will agree if writes stop, but no time limit is promised
Read-your-writes (read-after-write) After you save something, you always see it
Monotonic reads Later reads never show older data than earlier reads
Consistent prefix reads Writes are seen in the order they happened
Strong consistency Every read sees the latest write
Log sequence number A position in the replication log
Causal dependency One event happened because of another and must be seen after it
Sticky routing Sending the same user always to the same replica