Part 4 of 5 · 3 min read
Exercises
Easy
Easy 1: Read the table (15 minutes)
From the confusion table, say in two sentences what is wrong and which two cells to look at. Then list the six failing tickets and underline the word they share.
Done when: your description matches the clusters printed by clusters.
Easy 2: Write a hypothesis (15 minutes)
For a prompt you use (or an invented one), write three failure examples, find the common thread, and write the hypothesis as one sentence plus the single change you would test.
Done when: three failures, one hypothesis, one change.
Easy 3: Predict then check (20 minutes)
Before running it, predict the pass rate of Priority: bug > account > billing on the 30 cases, and which cases would regress. Then run it.
Done when: your prediction and the result are written side by side, with any surprise explained.
Hint
with bug > account > billing, a ticket with a bug word and an account word picks bug.
Medium
Medium 1: Find a better rule (60 minutes)
The best change in the demo still breaks t07 and t13. Extend the imitation model with one more kind of rule (for example, "if the ticket contains login and a money word, label by the first word in the sentence") and show a prompt change that fixes the six failures with no regressions. Explain why this is a hack and what a real model would do instead.
Done when: 30 of 30 cases pass and the limits are explained in a paragraph.
Hint
change the toy rules minimally and add tests for the new rule.
Medium 2: Distinctive words, better (60 minutes)
Improve distinctive_words to rank words by how much more common they are in failures (a ratio with smoothing), and to ignore words that are rare overall. Compare the top five with the old version on this data.
Done when: a test checks the ordering on a tiny hand-made report.
Hint
use (f + 1) / (p + 1) as a smoothed ratio and require f >= 2.
Medium 3: How many runs? (60 minutes)
With the noisy imitation at 15%, measure how the number of always_pass and flaky cases changes for k = 1, 3, 5, 10, 20, 50. Explain how many runs you need before the classification stops changing much.
Done when: a table of k against counts and a recommended k.
Hard
Hard 1: An automated bisect (120 minutes)
Write a tool that takes two stored prompt versions from the module 5 store, runs bisect_fields and prints a short report that names the culprit field and the regressed cases. Then extend it to bisect inside the constraints and examples lists, one item at a time.
Done when: a test finds a single bad rule inside a list of five.
Hint
a list-valued field can be bisected by adding items from the new list to the old one in turn.
Hard 2: Debug a real prompt (120 minutes)
Using a real model, run a 30-case set for a task of your choice, five runs each. Produce the confusion table or failure list, write a hypothesis, test two single changes, and report fixed and regressed cases. (I did not run this, so your results are the ground truth.)
Done when: a one-page debugging log with tables, and a note on which failures were flaky.
Hard 3: A prompt search loop (120 minutes)
Build a tiny automatic search: start from BASE, generate candidate changes from a list of rule snippets, evaluate each alone, keep any that fix cases without regressions, and repeat up to five rounds. Evaluate the final prompt on a held-out set of 15 tickets you write. Report the gap between the training and held-out scores.
Done when: a table shows rounds, training score and held-out score, and you comment on over-fitting.
Hint
accept a candidate only if fixed is non-empty and regressed is empty on the training cases.