Part 2 of 13 · 2 min read
A snapshot grows with the square of the run; a log grows in a straight line
Think of a save point in a game. If the game crashes, you restart from the last save, not from the very start.
In plain words: a checkpoint is a saved point from which a run can resume, a snapshot is the whole state written at that point, and an event log is an append-only list of changes from which state is rebuilt by applying them in order -- a snapshot resumes instantly because the whole state is already there, but writing the whole state on every step makes its total write volume grow with the square of the run length, while a log writes only the small change each step, which grows in a straight line but means replaying many events to resume.
A worked example. Over a 200-step run with 300-byte events, writing a full snapshot after every step totals 6,115 kB written and needs 0 events replayed on resume. A pure event log totals just 64 kB written but needs 100.5 events replayed on average. At 800 steps the snapshot total reaches 97,421 kB -- the quadratic growth is visible directly: going from 200 to 800 steps (4 times as long) multiplied the snapshot total by roughly 16, while the log's total only multiplied by about 4.
The key lines. From code/mini-durable/durable.py:
# from code/mini-durable/durable.py
def storage_cost(n_steps, mode, k=20, step_bytes=300):
"""Return (bytes written in total, events replayed on resume after a crash at a random step, averaged)."""
state, written, snaps = EMPTY, 0, [] # snaps: step index at which a snapshot exists
for i in range(n_steps):
ev = delta(i, step_bytes)
state = apply(state, ev)
if mode == "snapshot":
written += size(state); snaps.append(i)
elif mode == "log":
written += size(ev)
Read it as: in snapshot mode, size(state) is added every single step, and state itself keeps growing as messages accumulate. That is the mechanism behind the quadratic cost in one line. Each of the n snapshots is bigger than the last. So the total is proportional to roughly n times the average message count, which itself grows with n.
Checkpoint, snapshot, event log
A checkpoint is a saved point from which a run can resume. A snapshot is the whole state at that point. An event log (event sourcing) is an append-only list of changes; state is rebuilt by applying them in order.
Example: 6,115 kB for 200 snapshots against 64 kB for the same run's event log is the entire trade-off in two numbers -- a snapshot is simple and instant to resume from, a log is cheap to write, and a hybrid (Part 3's results) is how you buy some of each.