Part 9 of 12 · 2 min read
Mistakes that make a bug unreproducible
Common mistakes, and what the example does not do.
Mistakes that make a bug unreproducible
- Prompts edited in a dashboard with no history or review.
latestaliases in production.- Not logging the version with each trace.
- Assuming temperature 0 means identical outputs.
- Editing releases in place instead of publishing new ones.
- Unversioned retrieval indexes and tool schemas.
- In-flight runs switching versions mid-run.
- No retirement tracking, then a forced migration under pressure.
- Comparing outputs byte for byte instead of by evaluation.
What the example does not do
About this example What the example does not do
- The model is a stand-in: four canned variants, hash-chosen; real non-determinism has different causes and sizes.
- Assumed rates: 2 percent flip, 15 percent snapshot change, seeds always honoured (many APIs offer best-effort seeds or none).
- No real registry, prompt management product, tracing backend or deployment pointer system.
- No capture of external state: replay of tool results, retrieved documents and time is not modelled.
- Hashing JSON is simplified: no canonicalisation beyond sorted keys.
- No cost or latency versioning, and no retirement calendar integration.
- Optional real experiment: log 500 real requests with their full context, replay them against the same pinned snapshot at temperature 0 on different days, and measure exact and semantic agreement; then replay against a newer snapshot.
Check yourself
A model alias such as latest moves to a new snapshot overnight. What do you see?
Show answer
Behaviour changes with no change in your repo; in the stand-in model, replay agreement dropped from 95.6 percent to 84.4 percent. Pin an exact snapshot.
For the curious Temperature 0 is not a guarantee
With an assumed 2 percent chance per call of a different variant at temperature 0, replaying 500 logged requests matched 95.6 percent of the time; at temperature 0.7 with no seed, 36.6 percent; with a fixed seed, 100 percent, which holds only where the provider honours seeds. When a new prompt went live while 200 runs were mid-flight, all 200 unpinned runs mixed two versions; pinned runs, none. Compare with evals and tolerances, not byte equality.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Hash everything that decides behaviour | Code, prompt, model snapshot, settings, tools, index, guardrails. |
| Make releases immutable | Environments are pointers; rollback moves a pointer. |
| Pin runs to a release | Let old releases live until their runs finish. |
| Watch model retirement dates | Track them like any dependency deadline. |