Part 5 of 7 · 3 min read
Retention, lag and replay
A log is a large, fixed-size buffer on disk, and its readers can fall behind or go back.
Think of a notice board where new notices push the oldest off the end. If you are away too long, you miss the notices that have fallen off.
If a log only grows, the disk will fill. So DDIA explains that the log is divided into segments, and from time to time old segments are deleted or moved to cheaper archive storage. This is called retention. The log is effectively a bounded buffer that discards the oldest data when full, a circular or ring buffer, but because it is on disk, it is large.
DDIA makes a back-of-the-envelope calculation: a typical large drive then had 6 TB of capacity and 150 MB per second of sequential write speed. Written at full speed, it fills in about 11 hours.
| Step | Calculation | Result |
|---|---|---|
| Capacity in MB | 6 TB = 6,000,000 MB | 6,000,000 MB |
| Time to fill | 6,000,000 / 150 | 40,000 seconds |
| In hours | 40,000 / 3,600 | about 11 hours |
In practice, nobody writes at the full disk speed, so a log can usually hold several days or weeks of messages. The book adds that the log's throughput stays roughly constant however much history is kept, since every message is written to disk anyway. That differs from in-memory brokers, which are quick while queues are short and slow down when they spill to disk.
Lag. If a consumer cannot keep up, it falls behind the head of the log. The book says this can be monitored: measure how far behind each consumer is and raise an alert when it grows. The distance is called consumer lag.
Consumer lag
Consumer lag is the number of messages (or time) between the newest message in a partition and the consumer's current offset.
Example: the newest message in a partition has offset 10,000 and the consumer is at 9,400, so its lag is 600 messages.
Since the buffer is large, a human has time to fix the slow consumer before it is cut off from old data. If it does fall behind past the oldest retained segment, it misses messages. In the taxonomy of page 1, a log is a form of buffering with a large but fixed-size buffer. The book also stresses an operational plus: only the slow consumer is affected. Other consumers carry on, and a stopped consumer holds on to nothing but an offset. With a traditional broker, queues for dead consumers keep growing and must be deleted by hand.
Replay. Reading a log changes nothing, apart from the consumer's offset, and the offset belongs to the consumer. So a consumer can be started with yesterday's offsets, write its output somewhere else, and reprocess a day of messages. This can be repeated with changed code. DDIA says this makes log-based messaging resemble batch processing, with derived data cleanly separated from input data, and makes experiments and recovery from bugs easier.
A worked example. A bug in the billing consumer wrongly rounded amounts from 09:00 to 11:00. The topic keeps 7 days. The team fixes the code, starts a new consumer group at the offset of 09:00, writes to a new table, checks it, and swaps tables. The old data is still in the log, so nothing is lost.
Common mistake
Setting retention shorter than the longest outage a consumer could suffer. If retention is 6 hours and a consumer is down for 8, 2 hours of messages are gone for that consumer.
Check yourself:
- A topic receives 20 MB per second. How long does a 1 TB disk take to fill?
- What two actions does replay need that a deleting queue cannot offer?