Part 3 of 13 · 2 min read
Replaying a model call is not the same as re-running it
Think of rereading your own diary entry from a trip, versus asking yourself to redecide, today, what you would have ordered for dinner that night. Rereading reproduces exactly what happened. Redeciding gives you a plausible but different answer.
In plain words: replay means rebuilding state by re-applying recorded events, which only works if the function doing the applying is deterministic (the same inputs always give the same result) -- a live model call is never deterministic, so replaying a run by calling the model again produces a different run almost every time, while time travel (loading an earlier checkpoint to inspect or re-run from it) and fork (continuing from that checkpoint on a new branch, leaving the original untouched) both depend on replay actually being faithful to what happened.
A worked example. Simulating 500 independent 20-step runs, each with its own random seed, and comparing "replay from the recorded log" against "replay by calling the model again with a fresh seed": the fresh call diverges from the original recorded run in 99.6 to 100% of trials, even over just 20 steps. Replaying the recorded log instead reproduces the original run in 100% of trials, by construction, because it's re-applying the same recorded choices rather than drawing new ones.
The key lines. From code/mini-durable/durable.py:
# from code/mini-durable/durable.py
def replay(log, mode, n=None, seed=None):
"""mode 'recorded': re-apply logged results. mode 'live': call the model again with a new seed."""
if mode == "recorded":
return [e["choice"] for e in log]
fresh = run_live(len(log), seed)
return [e["choice"] for e in fresh]
Read it as: the "recorded" branch reads e["choice"] straight out of the stored log. The "live" branch calls run_live again with a seed that is explicitly a different one. The function's own two branches are a clean demonstration of the distinction. One reproduces and the other re-decides. Only the first one is actually replay.
Replay, deterministic, time travel, fork
Replay is rebuilding state by re-applying recorded events. Deterministic means the same inputs always give the same result; apply must be, and a live model call is not. Time travel is loading an earlier checkpoint to inspect or re-run from it. Fork is continuing from an earlier checkpoint on a new branch, leaving the original untouched.
Example: the near-100% divergence rate for a fresh model call is the exact reason a debugging workflow needs fork, not a live re-run -- to change one input and see what happens differently, you load the earlier checkpoint (time travel), branch from it (fork), and only then let the model make a new, deliberately different choice at that one point.