Part 6 of 7 · 4 min read
Living with lag
A way to decide how much consistency an application needs, and what it costs.
Living with lag
Think of a shop that tells customers that stock levels on the website are updated every few minutes. Most customers do not mind. The same shop would not accept a delay of minutes in its till, where a wrong total costs money.
The book gives a test for eventually consistent systems: think about how the application behaves if the lag grows to several minutes or even hours. If the answer is "no problem", good. If users would have a bad experience, the design needs a stronger guarantee such as read-your-writes. It warns that pretending replication is synchronous when it is asynchronous is a recipe for problems.
An application can add guarantees on top of the database, such as reading some data from the leader. But that is complex and easy to get wrong. The book says it would be better if developers could trust the database to do the right thing. Transactions exist for this reason: they let the database give stronger guarantees so that the application is simpler. Many distributed databases gave up transactions, claiming they cost too much in performance and availability. The book says there is some truth in that, but it is overly simple. It promises a more nuanced view later.
A procedure to choose:
- List the reads of the application.
- For each, ask: what would a user see if the data were 1 minute old? 1 hour old?
- Mark reads where staleness hurts (own profile after an edit, an order just placed, a bank balance).
- Give those reads the cheapest guarantee that fixes the problem: leader reads, session stickiness, or position tracking.
- Leave the rest on any follower.
A table of examples:
| Read | If stale | Guarantee needed |
|---|---|---|
| Home feed of other people's posts | Slight delay, unnoticed | None |
| Own post after publishing | Looks lost | Read-your-writes |
| Comments under a post, refreshed | A comment flickers | Monotonic reads |
| Chat messages | Reply before question | Consistent prefix |
| Account balance before payment | Wrong decision | Read from the leader |
| Product stock count in a list | Slightly wrong, fine | None |
| Stock count at checkout | Oversell | Read from the leader, or a transaction |
A worked example of the cost-benefit. A site sends 90 percent of reads to followers. Adding read-your-writes for own-profile reads raises leader reads by 3 percent of traffic. At 40,000 reads per second, that is 1,200 more leader reads per second. If the leader has 6,000 spare operations per second, the extra load is acceptable and the guarantee costs 20 percent of the spare capacity.
Interview angle
Answer in your head first, then open it.
What is replication lag, and what problems does it cause?
The delay before a follower shows a write. It causes lost own writes, time running backward and effects seen before causes.
How would you give a user their own writes?
Read recently changed data from the leader, or track the position of the user's last write and serve from a follower that has reached it. Mention multiple devices.
How do you stop a refresh from showing older data?
Route each user to the same follower, for example by hashing the user id.
Is eventual consistency acceptable here?
Walk through each read: what happens if it is a minute old. Give the strong guarantee only to reads that need it.
How could the lag grow large?
Bursts of writes, a follower recovering, network faults, a very slow query on the follower. Monitor lag and take lagging followers out of rotation.
Check yourself:
- Name two reads in a shop that can tolerate lag and two that cannot.
- Why is it better to rely on database guarantees than on special code in each application?
Words you will meet
| Word | Plain meaning |
|---|---|
| Replication lag | The delay between a write on the leader and on a follower |
| Eventual consistency | Replicas will agree if writes stop, but no time limit is promised |
| Read-your-writes (read-after-write) | After you save something, you always see it |
| Monotonic reads | Later reads never show older data than earlier reads |
| Consistent prefix reads | Writes are seen in the order they happened |
| Strong consistency | Every read sees the latest write |
| Log sequence number | A position in the replication log |
| Causal dependency | One event happened because of another and must be seen after it |
| Sticky routing | Sending the same user always to the same replica |