Part 4 of 5 · 3 min read
Exercises
Easy
Easy 1: Write ten cases (30 minutes)
For a task of your choice, write ten cases with expected results that code could check. Mark which two are the trickiest and why.
Done when: ten cases, each with an expectation you could turn into a check.
Easy 2: Read a comparison (15 minutes)
Run the demo and write three sentences for a reviewer explaining why v3 should not ship and what to change.
Done when: your message names the two regressed cases and the cause.
Hint
compare lists the regressed ids, and the diff shows the removed label.
Easy 3: Store by hand (20 minutes)
Save three versions with PromptStore, then edit one JSON file by hand and watch load refuse it. Explain in one sentence why that protection is useful.
Done when: you show the error message and your sentence.
Medium
Medium 1: A new check (60 minutes)
Add to Case a forbidden phrase list, and to check a rule that fails a reply containing any of them. Add three tests, including a case-insensitive match.
Done when: the new tests pass and the old ones still do.
Medium 2: Repeat runs (60 minutes)
Wrap the model in a noisy imitation that answers wrongly 10% of the time (seeded). Change run_suite to run each case 5 times and report the pass fraction per case. Choose a rule for when a case counts as passed (for example 4 of 5) and defend it.
Done when: a test shows a flaky case being reported with a fraction.
Hint
the reply for a flaky case differs between runs, so compare fractions, not single outcomes.
Medium 3: A markdown review report (60 minutes)
Write a function that turns compare and diff output into a short markdown report: prompt changes, fixed cases, regressions, gate result. Print it for v2 to v3.
Done when: the report contains all four parts and a test checks its text.
Hint
build the report as a list of lines joined with newlines.
Hard
Hard 1: Git-driven checks (120 minutes)
Write a script that finds which prompt files changed between two Git commits, runs the suite for old and new versions, and exits with a failing status if the gate fails. Run it in a temporary repository you create.
Done when: a scripted demo shows one passing and one failing change.
Hint
git diff --name-only A B -- prompts/ finds the changed files.
Hard 2: Real-model baseline (120 minutes)
Run your case set against a real model with each of three prompt versions, five times each. Report mean pass rate, spread, and cost (Level 2, module 6). State how many repeated runs you needed before the results stabilised. (I did not run this, so your numbers are the ground truth.)
Done when: a table and a short conclusion about noise.
Hard 3: Mine failures into cases (120 minutes)
Build a tool that takes a log of production replies marked bad by users, turns each into a draft Case (ticket plus expected label supplied by a human), and refuses to add duplicates. Show it growing the set from 12 to at least 20 cases and re-running the comparison.
Done when: a test covers the duplicate check and the report shows the larger set.
Hint
normalise ticket text (lower-case, collapse spaces) before comparing.