Part 5 of 6 · 4 min read
Exercises
Try each exercise before opening its hint. Where a real model is needed, any model you have access to will do.
Easy
Easy 1: Compounding errors (15 minutes)
Compute the end-to-end success rate for 5, 10 and 20 steps at 99%, 95% and 90% per-step accuracy. Put it in a 3 by 3 table.
Done when: the table is correct and you can say which cell is worst and why.
Hint
success is p ** steps; 0.95 to the tenth power is about 0.60.
Easy 2: Match the failure (15 minutes)
Name the failure mode in each case: (a) the model gives a different answer when you rephrase, (b) it agrees "you are right" when you say its correct answer is wrong, (c) it cites a paper that does not exist, (d) it ignores a rule stated at the top of a 60-page prompt, (e) it says a library's old version is the latest.
Done when: each has a mode and a one-line mitigation.
Easy 3: Run the detectors (20 minutes)
Run ungrounded_sentences and invented_numbers on a source sentence of your own and three answers: faithful, with an invented number, and with an invented claim without numbers.
Done when: you can state which detector caught which answer and which one slipped through.
Hint
use a number-free invented claim to see which detector is blind.
Medium
Medium 1: A real needle probe (60 minutes)
Write a 3,000-word filler text. Insert one made-up fact ("the access code for room 14 is 7391") at five depths. Ask a real model for the code each time, 5 trials per depth, and tabulate recall by depth.
Done when: you have a 5-row table, and a sentence saying whether your model shows a middle weakness at this length.
Hint
put the fact in a sentence that stands out from the filler, and ask for exactly the code, to keep scoring simple.
Medium 2: Wording lottery (60 minutes)
Choose a small classification task with 30 labelled examples. Write four phrasings of the instruction and score each on all 30 with a real model. Report accuracy per phrasing and the spread.
Done when: you can say whether the spread is bigger than you would expect from noise (use a bootstrap interval from module 5).
Medium 3: Make the grounding check better (60 minutes)
Extend ungrounded_sentences to compare against sentences in the sources rather than the whole word set, and add a check for named entities (capitalised words) not in any source. Add tests for two cases the current version misses, such as the negation in the blind-spot table.
Done when: the old tests still pass and the new tests demonstrate the improvement.
Hint
re.findall(r"\b[A-Z][a-z]+\b", text) finds capitalised words, but you must skip the first word of each sentence.
Hard
Hard 1: Probe the probe (90 minutes)
Write a test that runs needle_probe against simulated models with known recall curves (flat 90%, a middle dip, a decline toward the end) and asserts that the probe's output has the right shape for each, using a statistical tolerance. Explain how many trials you need for 3-point differences to be reliable.
Done when: three shape tests pass and you give a trial count with a reason.
Hint
the standard error of a proportion is sqrt(p * (1 - p) / n).
Hard 2: An answer verifier (120 minutes)
Build a function that takes (question, answer, sources) and returns a verdict supported, partly supported or unsupported, using sentence-level grounding, invented numbers, and self-consistency over several sampled answers (fake the samples). Test it on at least eight hand-written cases, including a wrong answer that overlaps the source heavily.
Done when: eight tests pass and you document the case your verifier gets wrong.
Hard 3: Cascade a pipeline (120 minutes)
Simulate a 10-step agent in which each step succeeds with probability 0.95. Compare four designs over 20,000 runs: no checks; a check after every step that catches 80% of errors and retries once; a single check at the end that catches 80% of errors; and a human review at the end that catches 100% but is applied only to runs flagged by a cheap check. Report end-to-end success and the expected number of checks.
Done when: you have a table of four designs and a recommendation with the reason.
Hint
model a retry as a second independent attempt at the same step.