Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-halluc, run pytest and python demo.py. For each of the seven sections of the demo output, write one sentence saying which control helped most and one thing it did not fix.
Done when: you have seven pairs of sentences that cite numbers from the output.
Hint
the demo prints one row per setting; compare "wrong" and "refused" together.
Easy 2: Classify the failure (15 minutes)
For each case, say whether it is mainly a context problem, an instruction problem, a checking problem or a decision problem, and name one control: (a) an agent quotes last year's price although the new price was retrieved too; (b) an answer states a feature that appears in no source; (c) the model answers confidently when asked about a product that is not in the sources; (d) a right passage in the middle of a long prompt is ignored.
Done when: you have four labels with one control each, and a note on any case that fits two labels.
Hint
(a) is about which passage went into the context; (d) is about where.
Easy 3: Read the curve (15 minutes)
Using the risk and coverage table in section 2.4, pick a threshold for each of: a children's homework helper, a bank's card-dispute assistant, an internal "where is the wiki page" bot. Give the coverage and risk you would accept for each and why.
Done when: you have three choices with a one-line reason each, and the bank's choice is stricter than the wiki bot's.
Hint
a stricter threshold gives lower risk and lower coverage.
Medium
Medium 1: Bias and order (45 minutes)
Change position_bias in experiments.py to 0, 0.75, 1.5 and 3.0. For each, measure run(plain, k=5, ...) with the three orderings (id order, best first, ends first). Present a 4 x 3 table.
Done when: you have the table and a sentence on when ordering stops mattering and when it matters most.
Hint
with no bias the three orderings should look alike.
Medium 2: More samples, better signal? (60 minutes)
Use risk_coverage with n = 3, 5, 9 and 15 samples. Plot or tabulate risk against coverage for each and say how much a larger n buys. Include the extra model calls for each choice.
Done when: you have four curves or tables, and you can say where extra samples stop being worth their cost.
Hint
the possible agreement values are multiples of 1/n, so small n gives few thresholds.
Medium 3: A smarter signal (60 minutes)
Self-consistency here compares whole answers. Add a second signal: the retrieval score of the best passage, or the top candidate's probability from distribution. Combine it with agreement and compare the combined rule with agreement alone on risk at equal coverage.
Done when: you have a comparison at (about) the same coverage, and a statement of whether the second signal added anything.
Hint
compare at matched coverage, not at the same threshold.
Hard
Hard 1: Real model, real curve (120 minutes)
Using a real model, sample five answers at temperature 0.7 for 40 questions (including 12 unanswerable ones). Measure agreement in a simple way (the same key fact appears in the answer), build the risk and coverage table, and compare it with the toy table. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table next to the toy one and at least two ways your model differs from the imitation.
Hint
write down the agreement rule before you look at the results.
Hard 2: Chain of verification (120 minutes)
Implement the draft, ask verification questions, answer them separately, revise loop with a real model for 20 questions. Count how often the revised answer is better, equal or worse than the draft, and the extra calls per question.
Done when: you have a better, equal, worse count and a recommendation for when the extra calls are justified.
Hint
keep the verification questions independent of the draft text where you can.
Hard 3: Needle in your haystack (90 minutes)
Using a real model and your own documents, place a single known fact at positions 1 to 10 in a ten-passage context, 20 trials per position, with a few unrelated passages as fillers. Plot accuracy by position and decide whether ordering the context matters for your model.
Done when: you have a 10-point curve, and a decision about context order supported by it. State the sample size and what the error bar could be.
Hint
with 20 trials per position, a difference under about 15 points is within noise.