CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make a bug unreproducible

Common mistakes, and what the example does not do.

Mistakes that make a bug unreproducible

  • Prompts edited in a dashboard with no history or review.
  • latest aliases in production.
  • Not logging the version with each trace.
  • Assuming temperature 0 means identical outputs.
  • Editing releases in place instead of publishing new ones.
  • Unversioned retrieval indexes and tool schemas.
  • In-flight runs switching versions mid-run.
  • No retirement tracking, then a forced migration under pressure.
  • Comparing outputs byte for byte instead of by evaluation.

What the example does not do

About this example What the example does not do
  • The model is a stand-in: four canned variants, hash-chosen; real non-determinism has different causes and sizes.
  • Assumed rates: 2 percent flip, 15 percent snapshot change, seeds always honoured (many APIs offer best-effort seeds or none).
  • No real registry, prompt management product, tracing backend or deployment pointer system.
  • No capture of external state: replay of tool results, retrieved documents and time is not modelled.
  • Hashing JSON is simplified: no canonicalisation beyond sorted keys.
  • No cost or latency versioning, and no retirement calendar integration.
  • Optional real experiment: log 500 real requests with their full context, replay them against the same pinned snapshot at temperature 0 on different days, and measure exact and semantic agreement; then replay against a newer snapshot.

Check yourself

A model alias such as latest moves to a new snapshot overnight. What do you see?

Show answer

Behaviour changes with no change in your repo; in the stand-in model, replay agreement dropped from 95.6 percent to 84.4 percent. Pin an exact snapshot.

For the curious Temperature 0 is not a guarantee

With an assumed 2 percent chance per call of a different variant at temperature 0, replaying 500 logged requests matched 95.6 percent of the time; at temperature 0.7 with no seed, 36.6 percent; with a fixed seed, 100 percent, which holds only where the provider honours seeds. When a new prompt went live while 200 runs were mid-flight, all 200 unpinned runs mixed two versions; pinned runs, none. Compare with evals and tolerances, not byte equality.

What you will use daily at work

Practice What it looks like
Hash everything that decides behaviour Code, prompt, model snapshot, settings, tools, index, guardrails.
Make releases immutable Environments are pointers; rollback moves a pointer.
Pin runs to a release Let old releases live until their runs finish.
Watch model retirement dates Track them like any dependency deadline.