Part 4 of 7 · 3 min read
Idempotent consumers
Make repeating a message harmless by making the operation safe to repeat.
Think of a light switch labelled "turn on" versus a button labelled "toggle". Pressing "turn on" twice leaves the light on. Pressing "toggle" twice puts it back off. The first is safe to repeat. The second is not.
DDIA defines the idea: an idempotent operation can be performed many times and has the same effect as performing it once.
Idempotent
An operation is idempotent if doing it several times leaves the system in the same state as doing it once.
Example: "set the balance of account 12 to 80" is idempotent. "Subtract 20 from the balance of account 12" is not, because every repeat subtracts again.
The DDIA book uses this pair. Setting a key in a key-value store to a fixed value is idempotent. Incrementing a counter is not.
Making an operation idempotent. Many useful operations are not naturally idempotent, but extra information can fix this. The Building Microservices book gives an example. A call that credits 100 points to account 1234 is not idempotent: send it twice and 200 points appear. If the call also says which purchase it is for (order 4567), and the service accepts only one credit per order, the call can be repeated without adding points twice.
The book stresses what idempotent means. It is the underlying business operation that must not repeat. It does not mean the call leaves no trace. You can still log that the call arrived and measure its time.
Three ways to make a consumer idempotent:
- Natural. Design the operation as "set to value" rather than "add".
- By unique id. The UDS book suggests giving each message a unique id and ignoring duplicates when reading. The consumer keeps a table of ids it has handled. For each message, it checks the table. If the id is there, it skips the work.
- By stored position. DDIA gives this for log consumers. Every message has a persistent, increasing offset. When writing a value to an external database, include the offset of the message that triggered the write. Then the consumer can tell whether an update was already applied.
One detail matters a lot: the check and the work must succeed or fail together. If the consumer does the work and crashes before recording the id, the repeat does the work again. The id and the result should be saved in the same local transaction (module 1 of Level 5).
A worked example. Message evt-9001 says "credit 100 points to account 1234".
| Step | Action | Database state |
|---|---|---|
| 1 | Transaction begins. Is evt-9001 in the table seen? No |
points = 0 |
| 2 | Add 100 to points. Insert evt-9001 into seen. Commit |
points = 100, seen = |
| 3 | The ack is lost. The message arrives again | |
| 4 | Is evt-9001 in seen? Yes. Skip the work and ack |
points = 100 |
The final state is the same as if the message had come once. The cost is one extra lookup and a table that grows. The table is trimmed by keeping ids only as long as a repeat could possibly arrive, for example 7 days (made up to show the idea).
Common mistake
Checking the id table, doing the work, and recording the id in two separate steps. A crash between them breaks the guarantee. Put the work and the id in one transaction, or use a unique constraint on the id so a second insert fails.
Idempotency also works across services. When a client calls a payment service, the client can send a unique request key, and the service stores the key with the result. A retry with the same key returns the stored result. This is a common industry practice, and it is general knowledge here, not drawn from the books above.
Check yourself:
- Which of these is idempotent: "append a line to a log", "set status to shipped", "add 1 to a counter"?
- Why must the id and the result be recorded in the same transaction?