Part 6 of 11 · 1 min read
How to build it so it holds up
- Give every factual claim a source (a passage, a tool result) and show it to the user or the checker.
- Extract and verify numbers, dates, names and identifiers by code against the source.
- Use an entailment model or a calibrated LLM judge for paraphrase, and measure its error rates (module 4).
- Allow and reward abstention: "not in the provided documents" is a correct answer.
- Use self-consistency selectively (high-stakes answers, no context available), with k of 3 to 5, and treat low agreement as a trigger for abstention or review.
- Measure on held-out claims from real traces (module 2), sliced by kind of claim.
- Log unsupported claims in production and sample them for review (module 8).