Part 8 of 11 · 2 min read
Mistakes that let a hallucination through
Common mistakes, and what the example does not do.
Mistakes that let a hallucination through
- Tuning detector rules on the test claims.
- Overlap-only checks that pass changed numbers.
- Treating a high consistency score as correctness.
- No abstention path, so the model must answer.
- Checking against retrieved text without checking the retrieval.
- Claims not extracted: scoring a whole paragraph as one unit.
- Ignoring the false-alarm cost of the checker.
- Fine-tuning to inject facts that change.
What the example does not do
About this example What the example does not do
- No real answers, and a hand-written, templated claim set of 22.
- Rules tuned on the test set; no held-out evaluation.
- No entailment model or LLM judge, so paraphrase is a false alarm by construction.
- A two-state model of knowledge in the consistency experiment; real uncertainty is graded.
- Independent samples; real samples at low temperature are correlated.
- Optional real experiment: collect 100 answers with their retrieved contexts, label each claim by hand, and compare overlap, number and negation checks, an entailment model and a judge on held-out claims.
Check yourself
A cheap word-overlap check passes an answer that changed 30 days to 90 days. Why?
Show answer
The words overlap almost completely; the changed figure is not seen. Numbers and identifiers need a code check against the source.
For the curious Self-consistency finds guesses, not confident errors
In the simulation, unknown facts were flagged 83 percent of the time with 2 samples, 97 with 3 and 100 with 5, and known facts raised no false alarms. A model that is confidently wrong gives the same answer every time, so consistency cannot flag it. The rules in this module were written while looking at the 22 test claims, so their 100 percent figures are optimistic.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Give each claim a source | Show the passage or tool result next to the answer. |
| Check numbers and names by code | Compare them with the source text. |
| Let the agent abstain | Not in the provided documents is a correct answer. |
| Use self-consistency selectively | 3 to 5 samples for high-stakes answers with no context. |