Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-faithful.
Done when: you report the three detectors' recall and false-alarm rate.
Easy 2: Add five claims (30 minutes)
Write five new claims of different kinds for the same context.
Done when: you report which the detector gets wrong and why.
Easy 3: Expected agreement (30 minutes)
A model guesses uniformly among 6 wrong answers on 4 samples.
Done when: you compute the chance that all four samples agree by luck.
Hint
6 times (1 over 6) to the power 4, which is 6 to the power minus 3.
Medium
Medium 1: Held-out check (3 hours)
Write 30 new claims without looking at the rules, then evaluate.
Done when: you report the drop from the tuned set.
Hint
do not look at is_supported while writing.
Medium 2: Date and name checks (3 hours)
Add a check for dates and capitalised names.
Done when: a test shows a changed date is flagged and an unchanged date is not.
Hint
extract dates with a regular expression and compare to the context.
Medium 3: Graded knowledge (3 hours)
Let the model know a fact with probability p per fact, varying by fact.
Done when: you plot recall against the share of low-p facts.
Hint
draw p per fact from a Beta distribution.
Hard
Hard 1: Real answers (8 hours)
Run a real agent on 50 questions with retrieved context and label every claim. (I did not run this, so your numbers are the ground truth.)
Done when: you report claim-level recall and false alarms for two detectors, on held-out claims.
Hint
label claims before running any detector.
Hard 2: Entailment model (6 hours)
Add an off-the-shelf entailment model to the claim check. (I did not run this.)
Done when: you report paraphrase handling and cost.
Hard 3: Abstention policy (5 hours)
Turn low support or low agreement into "I do not know". (I did not run this.)
Done when: you report the coverage lost and hallucinations avoided.
Hint
sweep the abstention threshold and plot coverage against error.