Part 7 of 12 · 1 min read
How to build it so it holds up
- Version everything that affects behaviour in a manifest; hash it; log the id on every request, trace, journal and eval run.
- Keep releases immutable and environments as pointers; make rollback a pointer move.
- Pin model snapshots; ban
latestin production; track retirement dates; test migrations as releases. - Store prompts in version control (or a registry with history), with templates validated for variables; review diffs.
- Log what you need to replay: inputs, retrieved context ids, tool results (or recorded stubs), sampling settings, seed.
- Evaluate by tolerance, not exact match; use multiple runs and statistics (Level 9).
- Replay a sample of production traffic against new releases before and after promotion; alert on agreement drops.
- Pin runs and queued jobs to their starting release; keep old releases deployable until they drain (modules 2, 3, 8).
- Version retrieval indexes and tool schemas like code; a re-embedded index is a release.
- Attach a model card and eval results to every release (Level 10 module 10).