CSE Roadmap

Core tier · about 4 hours · 11 parts

Part 7 of 11 · 2 min read

Answering hallucination questions in an interview

How do you detect hallucination in an agent's answers?

"I separate two situations. With a trusted context, I extract claims and check each against it: overlap for gross invention, code for numbers and negations, and an entailment model or a calibrated judge for paraphrase. Without a context, I use self-consistency: sample the answer several times and flag disagreement. On my small set, overlap alone caught 54 percent of false claims, because it passes changed numbers, and rules for numbers and negation raised that to 100 percent on the cases I tuned on, which is optimistic." Follow-up: "What do these miss?" Consistent wrong answers (at 60 percent consistent errors, 5-sample recall fell to 39 percent) and contexts that are themselves wrong.

Is self-consistency a good hallucination detector?

"It is a good detector of guessing: a model that knows a fact repeats it, one that guesses varies. In the example's simulation 5 samples flagged about 100 percent of guesses with no false alarms. But if the model errs consistently, the same answer on every sample, the agreement score looks perfect, so recall fell to 39 percent with 60 percent consistent errors. And it multiplies cost by k." Follow-up: "How would you reduce the cost?" Use it only on flagged or high-stakes answers.

How would you reduce hallucinations by design?

"Ground the answer in retrieved text with citations, constrain the answer to that context, let the model abstain, verify numbers and identifiers with tools, and put a human gate on high-stakes claims." Follow-up: "Which of those matters most?" Abstention: a model that says "I do not know" when it lacks support removes most of the remaining risk at the lowest cost.

How do you evaluate a hallucination detector?

"On a labelled set of claims drawn from real traces, with true and false claims, reporting recall, false-alarm rate and the breakdown by kind of claim, on held-out data. A small demonstration like the 22-claim one in this module is a warning: rules tuned on the same cases look perfect." Follow-up: "What would a good detector's false-alarm rate be?" Low enough that reviewers trust the flags; I would tie it to review capacity (Level 8 module 8).