CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 6 of 12 · 2 min read

What the simulation measured

Replay agreement under five conditions, and the manifest and pinning checks.

How to read the replay table: every row replays the same 500 logged requests; only the condition changes.

Replay of 500 logged requests (stand-in model) Identical outputs
Same snapshot, temperature 0 95.6%
Pinned snapshot, replayed later 95.6%
Temperature 0.7, no seed 36.6%
Temperature 0.7, fixed seed 100%
Alias silently moved to a new snapshot 84.4%

What to notice: the biggest single drop (95.6 to 36.6 percent) comes from sampling with no seed, not from anything about versioning. A seed is what turns that back into 100 percent. Pinning the snapshot only protects against the separate, smaller 84.4-percent alias-drift problem.

How to read the checks table: each row is an independent mechanical check, not a progression.

Check Result
Any one of 8 manifest fields changed release id changes
Extra space in a prompt release id changes (878347f787 to 3b31f86a1a)
Diff of two manifests lists only the changed fields
Promote, promote, rollback prod back at the first release; 2 manifests stored
200 runs in flight at a prompt change, unpinned / pinned 200 mixed / 0 mixed

What to notice: the last row is the starkest contrast in the whole table. Pinning turns "all 200 runs affected" into "zero runs affected," with the identical underlying prompt change. That is why pinning in-flight runs matters as much as pinning the model snapshot itself.

A provider's deprecation notice and retirement date are the clock this whole system races against. Once a snapshot is scheduled for retirement, migrating to a new one has to go through the same pipeline: a new manifest, a new release id, evals, and a canary. It should not happen by accident when the old snapshot simply stops responding.

Deprecation, retirement

Deprecation is a provider announcing that a model will be removed; retirement is the date after which it stops working.

Example: planning a snapshot migration as a deliberate release well before its retirement date, instead of discovering it has stopped responding.

{{svg:manifest|Components of a release, the release id, and the rules around it.}} {{svg:results|Replay agreement by condition, and in-flight version mixing.}}