Part 6 of 8 · 2 min read
Safety
Copies, acks and delivery guarantees.
Replication. Each partition has several copies (replicas) on different brokers, for example 3. One is the leader, which takes all writes. The others are followers, which copy from the leader. The layout is the replica distribution plan, made by an elected controller broker and stored in metadata.
In-sync replicas (ISR). A follower is in sync if it is not too far behind the leader. A slow follower is dropped from the ISR, so it cannot delay the others. A message is committed when all in-sync replicas have it, and only committed messages are readable.
Acknowledgments. The producer can choose when it hears "stored".
| Setting | Meaning | Effect |
|---|---|---|
| ACK = all | All in-sync replicas have it | Safest, slowest |
| ACK = 1 | The leader has it | Faster, but data is lost if the leader fails before copying |
| ACK = 0 | No wait, no retry | Fastest, loses data on failure, fine for metrics or logs |
A minimum ISR setting says how many replicas must be in sync for a write to be accepted. Higher is safer, and also less available. A worked example (made up). A partition has 3 replicas and a minimum ISR of 2, with ACK = all. If one broker is down, 2 are in sync and writes succeed. If two are down, only 1 is in sync, writes are refused, and the system favours safety over availability.
Delivery semantics.
- At-most-once. A message is delivered zero or one times. The producer does not retry, and the consumer commits the offset before processing. Messages can be lost. Suits metrics.
- At-least-once. A message is delivered one or more times. Producer uses ACK of 1 or all with retries, and the consumer processes first, then commits. Messages are not lost, but can repeat. This suits most uses, with consumers that handle duplicates (Level 10).
- Exactly-once. Each message has exactly one effect. It is hard, and costs performance. In practice it is at-least-once plus de-duplication. Suits money, such as payments and billing.
A worked example (made up). A consumer last committed offset 4. It processes 5, 6 and 7, then crashes before committing. A new consumer starts at offset 5. Messages 5, 6 and 7 are handled twice: 3 duplicates. If the consumer had committed before processing and then crashed, they would have been lost.
In-sync replica (ISR)
An in-sync replica is a copy of a partition that has kept up with the leader closely enough to count as safe.
Example: a follower 2 messages behind is in the ISR. One that is 5,000 behind is dropped until it catches up.
Check yourself: 1. What does the leader do that followers do not? 2. What is the cost of ACK = all? 3. How do duplicates arise in at-least-once?