Part 6 of 7 · 3 min read
Choosing a guarantee
Match the promise to what a lost or repeated message would cost.
Choosing a guarantee
Think of three jobs: counting website hits for a rough chart, charging a credit card, and sending a "welcome" email. A lost hit does not matter. A double charge does. A double welcome email is only annoying.
The right question for each message type is: what does a loss cost, and what does a repeat cost?
| Job | Cost of a loss | Cost of a repeat | Sensible choice |
|---|---|---|---|
| Rough traffic chart | Tiny | Tiny | At most once is fine |
| Exact order count | Wrong totals | Wrong totals | At least once plus idempotent consumer |
| Charge a card | Lost revenue, unhappy shop | Customer charged twice | At least once plus idempotency key |
| Welcome email | Missed email | Two emails | At least once, repeat is acceptable |
| Update a cache entry with a full value | Stale cache | None, it sets the same value | At least once, naturally idempotent |
The Fundamentals of Software Architecture book makes a related point about replays. In an event-driven design, replaying events is a way to recover, but it is complex "especially if the events/requests being replayed aren't idempotent". This links delivery guarantees to the safety of recovering after a failure: the more operations are idempotent, the more freely a team can re-run work.
A practical checklist for a design review:
- Write down every consumer and ask: what happens if this message arrives twice?
- For each non-idempotent consumer, add an id check or change the operation to "set".
- Ask: what happens if this message never arrives? For critical ones, add retries and a dead letter queue (page 1).
- Test by delivering each message twice in a test environment.
Interview angle
Answer in your head first, then open it.
Can a queue guarantee exactly-once delivery?
No. A lost acknowledgment forces a resend. The usual answer is at-least-once delivery plus idempotent processing, which gives an exactly-once effect.
How do you make a consumer idempotent?
Use a unique message id and a table of processed ids, saved in the same transaction as the work, or use "set" operations, or store the offset with the write.
At-most-once or at-least-once?
Ask what a loss costs against a repeat. Money and inventory want at-least-once with idempotence. Telemetry can accept loss.
What does exactly-once mean in Kafka-style streams?
Atomic commit of output, state and position inside the framework, so the visible effect happens once.
Check yourself:
- A fraud alert system must never miss an event but may alert twice. Which guarantee fits?
- Why do external side effects, such as emails, still need care in an exactly-once framework?
Words you will meet
| Word | Plain meaning |
|---|---|
| Delivery guarantee | A promise about how many times a message arrives |
| At most once | Zero or one delivery. May lose |
| At least once | One or more deliveries. May repeat |
| Exactly once | One effect per message |
| Effectively-once | Repeated work, but the effect shows once |
| Acknowledgment (ack) | The consumer telling the broker it is done |
| Idempotent | Safe to repeat, same result as once |
| Duplicate | A second copy of a message that was already handled |
| De-duplication | Dropping repeats, usually by message id |
| Checkpoint | A saved position or state to resume from |
| Offset | The position of a message in a log |
| Microbatching | Cutting a stream into tiny batch jobs |
| Atomic commit | Several effects happen together or not at all |
| Fencing | Stopping a node that was thought dead from acting |