Part 4 of 12 · 2 min read
Temperature 0 is a tendency, not a guarantee
Even pinning everything you can pin still leaves a small chance the model answers differently.
Think of asking the same barista for "the usual" on two different mornings. Most of the time you get an identical drink. On rare occasions a slightly different hand on the espresso machine gives you something subtly different. Nothing about your order changed.
In plain words: fixing the model snapshot and setting temperature to zero makes outputs mostly repeatable, but not perfectly — some residual randomness (batching, hardware) can still flip the answer on a small fraction of calls, and a sampled call needs a fixed seed to be exactly reproducible at all.
A worked example. Replaying 500 logged requests against the exact same pinned snapshot at temperature 0 produces an identical output only 95.6 percent of the time — not 100 — because of a small assumed chance of a different variant on each call. At temperature 0.7 with no seed, agreement drops to 36.6 percent, since sampling is genuinely random each time. Fixing a seed at the same temperature 0.7 restores agreement to 100 percent, because the random choice is now driven by a fixed number instead of fresh randomness.
The key lines.
# from code/mini-versioning/versioning.py
def model_call(snapshot, prompt, temperature, seed=None, t0_flip=0.02, rng=None):
base = int(h([snapshot, prompt]), 16) % len(VARIANTS)
if temperature == 0:
r = rng.random() if rng else 0.5
return VARIANTS[(base + 1) % 4] if r < t0_flip else VARIANTS[base]
r = random.Random(seed if seed is not None else rng.random())
return VARIANTS[(base + r.randrange(0, 3)) % 4]
Read it as: even the temperature-0 branch has a small random chance built in. The sampled branch's randomness is only tamed once a seed replaces the fresh random source. The practical consequence is that comparisons between releases need to be done through evals with tolerance. A strict byte-for-byte check will not work.
Replay
Re-running logged inputs against a given version to compare its outputs against what was originally recorded.
Example: the 500-request replay above, which measures agreement rather than assuming it.