Part 3 of 7 · 2 min read
Where duplicates and losses come from
A crash or a lost acknowledgment at the wrong moment decides which one happens.
Think of a cashier who takes your payment and prints the receipt. If the power fails after taking the money but before printing, you do not know whether the payment went through. You might pay again. Doubt about "did it happen?" is the root of every duplicate.
A message passes through three steps: the producer sends it, the broker stores it, the consumer processes it. At each step, an acknowledgment can be lost, so the sender cannot tell whether to repeat.
Consumer side. The UDS book describes a consumer that must delete a message from the channel when it is done, so no other consumer will read it. Two orders are possible:
| Order of steps | Crash at the bad moment | Result |
|---|---|---|
| Delete first, then process | Crash after delete, before processing | The message is lost for good (at most once) |
| Process first, then delete | Crash after processing, before delete | The message is read again (at least once) |
The book's conclusion: there is no such thing as exactly-once message delivery. The best a consumer can do is to simulate exactly-once message processing by requiring messages to be idempotent. The DDIA book gives the same reason in a different form: if the connection times out and the broker gets no acknowledgment, it must assume the message was not processed and deliver it again, although the message may have been fully processed and only the ack lost.
Log consumers. The UDS book describes a consumer that reads an ordered log and periodically saves a checkpoint of the last message it handled. If it processes a message and crashes before saving the checkpoint, it will read that message again on restart. DDIA says the same: messages processed after the last recorded offset are processed a second time.
Producer side. A producer sends a message and waits for the broker's confirmation. If the confirmation is lost (a network fault), the producer cannot know whether the broker stored the message. If it retries, there may be two copies in the broker. If it does not, the message may be lost. This is general reasoning that follows from the same unreliable-network facts as module 4 of Level 5, and not a quote from the books.
A worked example (made up). A queue delivers 1,000 payment messages. Each has a 1 in 100 chance of a crash between processing and acknowledging.
| Quantity | Calculation | Result |
|---|---|---|
| Messages processed twice | 1,000 x 1/100 | about 10 |
| Messages lost with at-least-once | none, since unacknowledged messages return | 0 |
| Messages lost with delete-first | about 10 crash after the delete | about 10 |
So at-least-once gives about 10 duplicates, and delete-first gives about 10 losses. For payments, a duplicate is fixable by checking, while a loss may go unnoticed. That is why at-least-once is the common default.
Check yourself:
- A consumer deletes a message after it processes it. Where must it crash to cause a duplicate?
- Why is a lost acknowledgment a problem for the sender?