Part 6 of 7 · 4 min read
Writing to two places: logs and the outbox
Avoid the dual write by writing once and letting others follow.
Writing to two places: logs and the outbox
Think of a shop assistant who must update the paper ledger and also the stock board. If they update the ledger and are called away, the board is wrong. Better: write once in the ledger and let another person copy from the ledger to the board.
A common need: when data changes in the database, also update a search index or cache. The DDIA book calls the naive way dual writes: the application writes to the database, then the search index. It names two problems:
- Races. Two clients update item X at once, client 1 to A and client 2 to B. The database sees A then B (final B). The search index sees B then A (final A). The two stay different forever, with no error.
- Partial failure. One write succeeds and the other fails. Making both succeed or fail is the atomic commit problem, which is expensive.
The UDS book gives the log-based answer. Instead of writing to the database and the index, the service appends one message to an ordered log, such as Kafka. That append is the atomic commit step. The database and the search index are asynchronous consumers that read the log in order and update at their own pace. Each consumer saves a checkpoint of the last message processed. After a crash it resumes from there. Because it may re-read a message, messages must be idempotent, for example by giving each a unique id and ignoring repeats. The book notes this is state machine replication again (module 8).
The outbox pattern. The outbox pattern is not described under that name in the books used for this course. The following is general industry knowledge, not from those books, and it applies the log idea inside a service's own database.
When a service changes data and also needs to publish an event, it writes both in one ordinary local transaction: the business row and a row in an outbox table that describes the event. A separate relay process reads new outbox rows and publishes them to the message broker, then marks them sent. If the relay crashes after publishing but before marking, it will publish again, so consumers must remove duplicates by event id. No distributed transaction is needed, because each step is a single-node transaction.
A worked example. An order service must save an order and tell the shipping service.
| Step | Action | If a crash happens here |
|---|---|---|
| 1 | One local transaction: insert order 501 and insert outbox row "OrderCreated 501" | Both rows are saved or neither |
| 2 | Relay reads the outbox row and publishes the event | Event is published on restart |
| 3 | Relay marks the row sent | If crash before this, the event is sent twice |
| 4 | Shipping service receives event id 9001 | It checks its "seen ids" and ignores a repeat |
Result: the order is never saved without the event eventually sent, and a repeated event does no harm. The cost: the event arrives a little late, so the system is eventually consistent.
| Approach | Atomic across systems? | Blocks? | Isolation | Typical use |
|---|---|---|---|---|
| 2PC / XA | Yes | Yes, if the coordinator fails | Yes | Short transactions in tightly managed systems |
| Saga | Eventually, with compensation | No | No | Long business processes across services |
| Log or outbox | Eventually | No | No | Keeping derived stores and other services in step |
Interview angle
Answer in your head first, then open it.
How do you do a transaction across two services?
Prefer to avoid it. Use a saga with compensations, idempotent steps and a persisted state machine. 2PC if both sides support it and the transaction is short.
What goes wrong with 2PC?
A coordinator crash leaves participants in doubt holding locks. It blocks.
What is a saga?
Local transactions plus compensating steps. No isolation.
How do you update the database and publish an event safely?
The outbox pattern: one local transaction writes both, a relay publishes, consumers de-duplicate.
Exactly-once?
Really at-least-once plus idempotent processing.
Check yourself:
- Why is a dual write unsafe even when no error occurs?
- In the outbox pattern, why must consumers de-duplicate?
Words you will meet
| Word | Plain meaning |
|---|---|
| Atomic commit | All nodes commit or all abort |
| Distributed transaction | A transaction touching more than one node or system |
| Two-phase commit (2PC) | Prepare, then commit or abort, driven by a coordinator |
| Coordinator, transaction manager | The process that runs the 2PC steps and records the decision |
| Participant | A node that does part of the transaction |
| Prepare, vote | The first phase, where each participant promises or refuses |
| Commit point | The moment the coordinator durably records its decision |
| In doubt | Voted yes but has not heard the decision |
| Blocking protocol | Cannot move on until a failed part recovers |
| XA | A standard for 2PC across different technologies |
| Saga | Local transactions plus compensating transactions |
| Compensating transaction | A new transaction that undoes a previous one |
| Orchestrator | The coordinator of a saga |
| Semantic lock | A flag on data to say a saga is still working on it |
| Dual write | Writing the same change to two systems separately |
| Outbox pattern | Saving an event in the same local transaction as the data and publishing it later |
| Idempotent | Doing it twice has the same effect as once |