Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-trajectory.
Done when: you explain each cell in the pass-rate table in one phrase.
Easy 2: Add an invariant (30 minutes)
Add "send_email only after refund".
Done when: a test shows a run that emails first is flagged.
Hint
INVARIANTS is a list of (before, after) pairs.
Easy 3: Milestone score (30 minutes)
Compute the milestone score for ["verify_identity", "refund"] by hand.
Done when: it matches milestone_score.
Hint
the longest common subsequence of the two lists, divided by 4.
Medium
Medium 1: Argument checks (3 hours)
Represent steps as (tool, arguments) and add a check that the refund amount equals the order total.
Done when: a wrong-amount run is caught by the trajectory scorer, and you list one case it still misses.
Hint
compare the argument in the refund call with the order record.
Medium 2: Several references (2 hours)
Let a task have two acceptable reference paths.
Done when: the exact-path scorer passes both and you compare false failures with a rules-based scorer.
Hint
any(exact_match(p, r) for r in refs).
Medium 3: Budget calibration (2 hours)
Set the step budget from the 95th percentile of known-good runs.
Done when: you report the budget and the false-failure rate on good runs.
Hint
take the percentile over step counts of runs that pass the outcome check.
Hard
Hard 1: Real traces (6 hours)
Score 50 real runs outcome-only and with rules. (I did not run this, so your numbers are the ground truth.)
Done when: you report how many outcome passes break an invariant and what the worst violation was.
Hint
parse the trajectory from your trace store (module 8).
Hard 2: LLM judge for path quality (6 hours)
Add a judge for "was each tool call justified?" and check it against human labels. (I did not run this.)
Done when: you report agreement and the failure cases.
Hint
keep the judge's rubric short and test it on known-bad steps.
Hard 3: Online invariant monitor (5 hours)
Run the same invariants on a stream of live traces. (I did not run this.)
Done when: you report the violation rate, alerts and false alarms.
Hint
batch by hour and alert on rate changes, not single events.