Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-versioning.
Done when: you report replay agreement for five conditions.
Easy 2: Write your manifest (30 minutes)
List every component that affects your agent's behaviour and where each is stored today.
Done when: each has a version or a plan for one.
Hint
include the retrieval index, the tool schemas and any config flags.
Easy 3: Find your aliases (30 minutes)
Search your code and config for model aliases.
Done when: each is replaced by a pinned snapshot or justified.
Medium
Medium 1: Release ids in traces (3 hours)
Add the manifest id to your tracing spans (Level 9 module 8).
Done when: you can chart error rate by release id.
Hint
put the id in the root span and in the logs.
Medium 2: Prompt registry (4 hours)
Move prompts to files with strict variables and tests.
Done when: a missing variable fails CI and a prompt change appears in the release diff.
Hint
render templates with a strict formatter and test unused variables too.
Medium 3: Pin runs (3 hours)
Store the release id in each job and resume with it.
Done when: a mid-run prompt change does not affect running jobs.
Hard
Hard 1: Real replay (8 hours)
Replay logged production requests against a pinned snapshot on different days and against a newer snapshot. (I did not run this, so your numbers are the ground truth.)
Done when: you report exact and semantic agreement and the cases that changed.
Hint
compare with an evaluator or judge, not just string equality.
Hard 2: Migration (8 hours)
Migrate to a newer model snapshot through your pipeline. (I did not run this.)
Done when: evals, canary and rollback all passed and you recorded the release diff.
Hard 3: Incident drill (6 hours)
Introduce an unannounced change (alias, tool schema or index) and find it using only manifests and traces. (I did not run this.)
Done when: you identify the cause and time how long it took.