Definitions

Definitions

The manifest

Results

1. Reproducibility is a property you design for, with manifests, pinned snapshots and logged inputs.
2. Agreement under replay is a monitoring signal; drops with no code change mean the model, index or tools moved.
3. Rollback should be a pointer move.
4. Runs need to know which release they belong to.

Design rules

• Version everything that affects behaviour in a manifest; hash it; log the id on every request, trace, journal and eval run.
• Keep releases immutable and environments as pointers; make rollback a pointer move.
• Pin model snapshots; ban latest in production; track retirement dates; test migrations as releases.
• Store prompts in version control (or a registry with history), with templates validated for variables; review diffs.
• Log what you need to replay: inputs, retrieved context ids, tool results (or recorded stubs), sampling settings, seed.
• Evaluate by tolerance, not exact match; use multiple runs and statistics (Level 9).
• Replay a sample of production traffic against new releases before and after promotion; alert on agreement drops.
• Pin runs and queued jobs to their starting release; keep old releases deployable until they drain (modules 2, 3, 8).
• Version retrieval indexes and tool schemas like code; a re-embedded index is a release.
• Attach a model card and eval results to every release (Level 10 module 10).

Interview questions, with the follow-ups

Common mistakes

• Prompts edited in a dashboard with no history or review.
• latest aliases in production.
• Not logging the version with each trace.
• Assuming temperature 0 means identical outputs.
• Editing releases in place instead of publishing new ones.
• Unversioned retrieval indexes and tool schemas.
• In-flight runs switching versions mid-run.
• No retirement tracking, then a forced migration under pressure.
• Comparing outputs byte for byte instead of by evaluation.

What the example does not do

• The model is a stand-in: four canned variants, hash-chosen; real non-determinism has different causes and sizes.
• Assumed rates: 2 percent flip, 15 percent snapshot change, seeds always honoured (many APIs offer best-effort seeds or none).
• No real registry, prompt management product, tracing backend or deployment pointer system.
• No capture of external state: replay of tool results, retrieved documents and time is not modelled.
• Hashing JSON is simplified: no canonicalisation beyond sorted keys.
• No cost or latency versioning, and no retirement calendar integration.
• Optional real experiment (I did not run this): log 500 real requests with their full context, replay them against the same pinned snapshot at temperature 0 on different days, and measure exact and semantic agreement; then replay against a newer snapshot.

For the curious Temperature 0 is not a guarantee

With an assumed 2 percent chance per call of a different variant at temperature 0, replaying 500 logged requests matched 95.6 percent of the time; at temperature 0.7 with no seed, 36.6 percent; with a fixed seed, 100 percent, which holds only where the provider honours seeds. When a new prompt went live while 200 runs were mid-flight, all 200 unpinned runs mixed two versions; pinned runs, none. Compare with evals and tolerances, not byte equality.

What you must be able to say (one page)

• An agent's behaviour is decided by more than code. Code, prompts, the model (and its snapshot), sampling settings, tool schemas, retrieval index, guardrail rules and configuration all change what it does. If you cannot say exactly which combination served a request, you cannot reproduce a bug, compare releases or roll back safely.
• Define a release manifest and hash it. My Manifest holds code sha, exact prompt text, model snapshot, temperature, seed, tool schema hashes, index version and guardrail set; the release id is a hash of all of it, recorded in every trace, journal record and eval run (modules 3 and 8, Level 9). Manifests are immutable values; changing any one field (I tested all eight) changes the id, even an extra space in a prompt (id 878347f787 became 3b31f86a1a), and a diff between two manifests lists exactly the field that changed. That diff answers "what changed since yesterday?" in seconds.
• Pointers, not copies. A registry stores immutable manifests; prod is a pointer to a release id. Promote moves the pointer, rollback moves it back (tested: after rolling back, prod pointed at the original id and both manifests were still stored). Never edit a release in place.
• Pin model snapshots, avoid aliases. A provider alias such as latest or default can start pointing to a new snapshot with no change in your repo; behaviour shifts silently. In my stand-in model, an alias that moved to a new snapshot changing 15 percent of prompts dropped replay agreement from 95.6 percent to 84.4 percent. Providers also retire snapshots on published schedules, so track retirement dates (check your provider's current deprecation policy) and plan migrations as releases with evals (Level 9 module 7).
• Temperature 0 is not a guarantee. You can fix the sampling settings (temperature, top-p, top-k) and the seed, but even then results may not be fully reproducible. In my stand-in model with an assumed 2 percent chance per call of a different variant at temperature 0 (batching or hardware non-determinism), replaying 500 logged requests against the same snapshot matched 95.6 percent of the time (two independent 2 percent flips); at temperature 0.7 with no seed 36.6 percent; with a fixed seed 100 percent, which holds only where the provider honours seeds. Compare behaviour through evals and tolerances, not byte equality: re-run a test 3 to 5 times and check that results stay within acceptable bounds.
• Prompts are code: keep them in the repo (or a registry), review changes, give them identifiers, render them with strict variables (my render raises on missing or unused variables), and test them with the regression suite (module 8); an evaluation set acts as a living specification that supports reproducibility and regression detection. Template details matter across model versions: chat templates can change between versions of the same model, and a stale template causes hard-to-find bugs.
• Pin each run to its release. If a new prompt goes live while 200 runs are mid-flight at step 3 of 6, all 200 unpinned runs mixed two versions; pinned to their starting release, none did. Store the release id in the run's journal (module 3) and let old releases stay deployable until their runs finish; do the same for queued jobs (module 2).
• Record versions per metric. Track, for example, failed tasks per model version. With release ids in traces you can chart quality, cost and latency by release and compare shadow or canary versions (Level 9 module 9).
• Ship a compliance bundle with each version: model card, datasets and eval results, published to a model registry with an updated datasheet per version (Level 10 module 10).
• Honest limits: the model is a stand-in with assumed non-determinism; no real provider, registry, tracing system or prompt management tool.