Part 6 of 12 · 2 min read
What the simulation measured
Replay agreement under five conditions, and the manifest and pinning checks.
How to read the replay table: every row replays the same 500 logged requests; only the condition changes.
| Replay of 500 logged requests (stand-in model) | Identical outputs |
|---|---|
| Same snapshot, temperature 0 | 95.6% |
| Pinned snapshot, replayed later | 95.6% |
| Temperature 0.7, no seed | 36.6% |
| Temperature 0.7, fixed seed | 100% |
| Alias silently moved to a new snapshot | 84.4% |
What to notice: the biggest single drop (95.6 to 36.6 percent) comes from sampling with no seed, not from anything about versioning. A seed is what turns that back into 100 percent. Pinning the snapshot only protects against the separate, smaller 84.4-percent alias-drift problem.
How to read the checks table: each row is an independent mechanical check, not a progression.
| Check | Result |
|---|---|
| Any one of 8 manifest fields changed | release id changes |
| Extra space in a prompt | release id changes (878347f787 to 3b31f86a1a) |
| Diff of two manifests | lists only the changed fields |
| Promote, promote, rollback | prod back at the first release; 2 manifests stored |
| 200 runs in flight at a prompt change, unpinned / pinned | 200 mixed / 0 mixed |
What to notice: the last row is the starkest contrast in the whole table. Pinning turns "all 200 runs affected" into "zero runs affected," with the identical underlying prompt change. That is why pinning in-flight runs matters as much as pinning the model snapshot itself.
A provider's deprecation notice and retirement date are the clock this whole system races against. Once a snapshot is scheduled for retirement, migrating to a new one has to go through the same pipeline: a new manifest, a new release id, evals, and a canary. It should not happen by accident when the old snapshot simply stops responding.
Deprecation, retirement
Deprecation is a provider announcing that a model will be removed; retirement is the date after which it stops working.
Example: planning a snapshot migration as a deliberate release well before its retirement date, instead of discovering it has stopped responding.
{{svg:manifest|Components of a release, the release id, and the rules around it.}} {{svg:results|Replay agreement by condition, and in-flight version mixing.}}