CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Version everything that affects behaviour in a manifest; hash it; log the id on every request, trace, journal and eval run.
  • Keep releases immutable and environments as pointers; make rollback a pointer move.
  • Pin model snapshots; ban latest in production; track retirement dates; test migrations as releases.
  • Store prompts in version control (or a registry with history), with templates validated for variables; review diffs.
  • Log what you need to replay: inputs, retrieved context ids, tool results (or recorded stubs), sampling settings, seed.
  • Evaluate by tolerance, not exact match; use multiple runs and statistics (Level 9).
  • Replay a sample of production traffic against new releases before and after promotion; alert on agreement drops.
  • Pin runs and queued jobs to their starting release; keep old releases deployable until they drain (modules 2, 3, 8).
  • Version retrieval indexes and tool schemas like code; a re-embedded index is a release.
  • Attach a model card and eval results to every release (Level 10 module 10).