Part 8 of 12 · 2 min read
Answering versioning questions in an interview
How do you reproduce yesterday's behaviour?
"I record a release manifest id on every request: code sha, exact prompts, pinned model snapshot, sampling settings, tool schemas and index version. I can then check out that release, replay the logged inputs and context, and compare. Temperature 0 is not a guarantee, so I judge by evals with tolerances; in the example's toy model even the same snapshot at temperature 0 matched only 95.6 percent." Follow-up: "What if the provider changed the model?" That is why I pin snapshots; aliases moved agreement down to 84 percent in the example's test with no code change.
How do you roll back a prompt?
"Prompts are part of an immutable release; production is a pointer, so rollback moves the pointer to the previous release id. In-flight runs stay pinned to the release they started with." Follow-up: "What about queued jobs?" They carry the release id or are compatible with both versions during the rollout (module 8).
Why not just use latest?
"It makes behaviour depend on someone else's deployment schedule: it can change without any commit, so a regression has no diff to find. I pin snapshots and migrate deliberately with evals, tracking retirement dates." Follow-up: "Isn't pinning stale?" Yes, so migration is a planned release, not an accident.
How do you find what changed when quality dropped?
"Diff the manifests of the good and bad releases, check agreement on replayed traffic by release, and look at quality metrics per release id. If nothing in my manifest changed but quality did, the cause is outside it: data, traffic mix, a tool or the provider." Follow-up: "What if the model is the same but a tool changed?" The tool schema hash is in the manifest, and tool behaviour changes show up in the recorded tool results.