Part 5 of 7 · 3 min read
Starting a new copy: snapshots and compaction
A new derived system needs the old data as well as the changes from now on.
Think of joining a book club halfway through the year. If you only get next week's notes, you lack the story so far. You need the whole book up to a certain page, and then the notes from that page onward.
DDIA explains the problem. If you keep every change ever made to a database, replaying the log rebuilds the entire database. But keeping all changes forever takes too much disk space, and replaying takes too long, so the log is trimmed. A new full-text index needs a full copy of the data, since items that were not recently updated would be missing.
Option 1: snapshot plus log. Take a consistent snapshot of the database (as when setting up a follower in Level 4). The snapshot must correspond to a known position in the change log, so you know where to start applying changes after the snapshot is loaded. Some CDC tools include this. Others leave it manual.
Option 2: log compaction. In a log-structured storage engine (Level 2), the engine looks for records with the same key, throws away the duplicates and keeps only the most recent. A delete is recorded with a special null value called a tombstone, which removes the key at compaction. The space needed depends only on the current contents, not on the number of writes ever made.
The same idea works on the change stream. If every change has a primary key and each event carries the whole new version of the record, it is enough to keep the latest event per key. Then a new consumer can start from offset 0 of the compacted topic and read the whole log. It is guaranteed to contain the latest value of every key, and so gives a full copy of the database without a new snapshot. Kafka supports this feature. It means the broker can act as durable storage, not just transient messaging.
Log compaction
Log compaction is a background process that removes older records with the same key from a log, keeping only the latest one for each key.
Example: a topic holds (product 41, price 5), (product 42, price 9), (product 41, price 7). After compaction it holds (product 42, price 9), (product 41, price 7).
Tombstone
A tombstone is a record with a key and an empty value, which marks the key as deleted. Compaction removes the key and the tombstone itself after a while.
Example: (product 42, empty) means product 42 was deleted.
A worked example. A table has 1,000,000 products, and changes arrive at 50 per second.
| Without compaction | With compaction | |
|---|---|---|
| Events per day | 50 x 86,400 = 4,320,000 | Same arrive |
| Events kept after 30 days | about 129,600,000 | At most about 1,000,000 plus recent ones |
| Time to rebuild by reading all | Grows forever | Bounded by table size |
(Both rows are rough, using made-up numbers.) With compaction the log stays about the size of the table, and a new index can read it from the beginning.
Compaction does not suit event sourcing (page 4). There the later event does not replace the earlier ones, so every event is needed. It also fails for events whose meaning needs history. For CDC, where each event carries the complete new record, compaction fits.
Check yourself:
- Why must a snapshot be tied to a position in the change log?
- How can a compacted log replace a snapshot?