Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-reflect, run pytest and python demo.py. Report for each of the four strategies the final valid share and the model calls per job, and say which two results you find most surprising.
Done when: you give a table of both numbers and a one-sentence reason for each surprise.
Hint
the cheapest strategy to run is not the one that spoils drafts.
Easy 2: Find the outside check (20 minutes)
For each task, write one check a program could run instead of asking the model to review: (a) an SQL query that must return one row; (b) a summary that must keep all dollar amounts from the source; (c) a translation that must keep placeholders like {name}; (d) a meeting time that must not clash with a calendar.
Done when: each check is described precisely enough that someone could code it and says what message it returns.
Hint
(a) run it against a copy of the data; (d) compare time ranges.
Easy 3: Spot the false alarm (15 minutes)
A model reviews the correct text "We have refunded $23 for the wait." and says "The tone is cold and the refund should mention the order id." Explain what could go wrong if the writer obeys, and what instruction to the critic would reduce this.
Done when: you describe one specific damage and one change to the critic's instructions (for example "list only violations of this rule list, quoting the text").
Hint
the damage is that a correct part gets edited.
Medium
Medium 1: Add a seventh check (60 minutes)
Add a check: the notice must mention the carrier name (a new field on Job). Add the slip to the writer, the message, and tests. Re-run the table in section 3.
Done when: the new check is tested, and you report the new first-draft valid rate and whether the ranking of strategies changed.
Hint
add the flaw name to CHECKS, BASE_FLAW, render and message together.
Medium 2: Stop when nothing changes (60 minutes)
Add a "no progress" rule to revise from checks: stop if the same set of failed checks appears twice in a row, and return the best draft seen. Add a test with a writer whose fix chance is zero.
Done when: the test shows it stops at round 2 instead of using all rounds, and the returned draft has the fewest failed checks seen.
Hint
compare sorted tuples of failed check names.
Medium 3: Make the critic quote evidence (90 minutes)
Change the toy self-critic so it only raises a problem when it can quote the exact text that breaks the rule (model this as: false alarms drop to 3% and detection drops to 45%). Compare against the original critic in the grid of section 4, and explain the trade.
Done when: you have both results and say in which region of the grid the quoting critic is better.
Hint
write down both rates, run the grid, look where the cells cross 47%.
Hard
Hard 1: A real producer and critic (150 minutes)
Pick a real writing task with at least five rules. Build the three variants (self-critique, a critic with a different prompt or model, and rule checks in code) with a real model, run 50 jobs, and measure final pass rate, calls and spoiled drafts. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table like section 3 from a real model and a paragraph on the spoiled drafts.
Hint
use the same writer instructions in all three variants so only the critic changes.
Hard 2: Lessons with decay (150 minutes)
Extend the lessons idea so each lesson has a confidence that rises when it prevents a slip and falls when the slip happens anyway, and lessons below a threshold are dropped. Test on 200 jobs where the writer's slip rates change halfway.
Done when: you show that the decaying memory recovers faster after the change than permanent lessons, and report both curves.
Hint
store (lesson, confidence); on each job update only lessons that apply.
Hard 3: Cost-aware reflection (150 minutes)
Build a policy that decides per draft whether to look again, using cheap signals (cheap checks passed or failed, draft length, how confident the writer says it is). Compare against always reflecting and never reflecting on quality and cost.
Done when: your policy matches the quality of always-reflecting at lower cost on a held-out set, or you show honestly why it cannot.
Hint
log the cheap signals with the final pass or fail first; fit the policy on one half, test on the other.