CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 3 of 7 · 2 min read

Why dual writes fail

Writing to two systems from application code can leave them different without any error.

Think of two clerks who each keep a copy of the price list. A manager tells clerk A "price is 5" and then clerk B "price is 5". Meanwhile another manager tells B "price is 7" first, and A "price is 7" second. The last word at A is 7, and at B it is 5. No one made a mistake, and the lists differ.

DDIA describes dual writes: the application code writes to each system itself. First it writes to the database, then it updates the search index, then it clears the cache entry. The book finds two serious problems.

Race condition. Two clients update item X at nearly the same time. Client 1 wants A and client 2 wants B. The database receives A and then B, so its final value is B. The search index receives B and then A, so its final value is A. The two systems are now different for good, and nothing failed.

Step Database Search index
Client 1 writes A to the database A
Client 2 writes B to the database B
Client 2 writes B to the index B B
Client 1 writes A to the index B A

The book adds that unless there is some concurrency detection, such as version vectors (Level 4), you will not even notice.

Partial failure. One write succeeds and the other fails. This is a fault-tolerance problem. Making both succeed or both fail is atomic commit (Level 5, module 9), which is expensive. The UDS book works through this with a product catalog and a search index. If the service updates the database and crashes before updating the index, the system is inconsistent. Two-phase commit would fix it, but the search index may not support the XA standard, the service would block if the index were down, and eventual consistency is enough for this use anyway.

Dual write

A dual write is an application writing the same change separately to two or more systems, without a shared transaction or a shared order.

Example: the code saves a product to the database and then calls the search index to save it again. If the second call fails, the two differ.

DDIA sees the root cause: there is no single leader that decides the order of writes. The database may have its own leader and the search index its own, and neither follows the other, so conflicts can occur. The fix is to have one leader, the database, and make the other systems followers.

Check yourself:

  • In the table above, which write order at the index causes the permanent difference?
  • Why is two-phase commit a poor fit for updating a search index?