Part 4 of 8 · 2 min read
Storage
Append, segment, batch.
Where should messages be stored? A relational or NoSQL database can store them, but it is hard to make one handle heavy writes and heavy reads at this scale, and it becomes a bottleneck. The book recommends a write-ahead log (WAL): a plain file where new entries are only appended to the end. Many systems use the idea, such as the redo log in MySQL.
Why this works: it follows a sequential pattern, always writing at the end and reading in order. Spinning disks are fast at this. Several hundred MB per second is possible, and disks are cheap and large. The operating system also keeps recent data in memory, which speeds up reads further. Random access is what makes disks slow, and the log avoids it.
Segments. A file cannot grow for ever. So a partition is cut into segment files. New messages go only to the active segment. When it reaches a size limit, it is closed and a new one starts. Old segments are only read, and are deleted when they pass the retention limit, here two weeks.
Message layout. The queue is fastest when the producer, the broker and the consumer all use one format, so no one has to rewrite the message and copy it. Fields: key, value (the payload), topic, partition, offset, timestamp, and a checksum (CRC) to detect damage.
Batching. Small writes are the enemy of throughput. Producers send messages in batches, the broker writes larger blocks, and consumers fetch batches. This saves network round trips and disk operations. There is a trade-off: a bigger batch means higher throughput but a longer wait to fill it. A smaller batch gives lower delay and less throughput.
A worked example (made up). A producer sends 1,000 messages of 1 KB. As single requests, this is 1,000 round trips. In batches of 100, it is 10. Each round trip also costs a little time, so the batch is much cheaper overall, even though the first message waits for the batch to fill.
Write-ahead log (WAL)
A write-ahead log is a file that is only ever appended to, so every write is sequential and old entries never change.
Example: a partition's segment file holds messages 1 to 100,000 in the order they arrived.
Check yourself: 1. Why are disks fast for a log but slow for random access? 2. Why split the log into segments? 3. What is the trade-off in batch size?