Part 8 of 11 · 2 min read
Mistakes that produce a fluent, wrong report
Common mistakes, and what the example does not do.
Mistakes that produce a fluent, wrong report
- One retrieval for a multi-part question, then confident synthesis.
- Citing a page that does not contain the claim.
- Hiding conflicts to make the answer look clean.
- Trusting forum posts or very old pages without dates.
- Treating page text as instructions.
- Counting repeated claims as independent confirmation.
- No "not found" path, so the agent invents.
- No budget, so one hard question burns the day's spend.
- Evaluating only the prose, not the claims and sources.
- Dropping source dates and URLs from the report.
What the example does not do
About this example What the example does not do
- No model and no web: the planner is rules, the writer is a script and the corpus is invented.
- Claims are structured (
CLAIM entity | attribute | value), which makes verification exact; real pages need extraction, matching and judgement, which are the hard part. - The corpus is tiny, so retrieval is easy; real corpora have duplicates, near-duplicates, paywalls and noise.
- The injection defence is a regular expression, easy to evade; it demonstrates the structure (data, not instructions), not a robust filter.
- No citation-laundering detection, no source-origin tracing, no handling of numbers with units or ranges.
- The ranking rule (kind, then date) is a simple choice; recency can be wrong when an old primary source is the origin.
- Optional real experiment: pick ten factual questions with known answers in a domain you know, run a real model with search, and score coverage, correctness, citation precision (open each cited page) and cost. Then add a verifier and repeat.
Check yourself
Why plan the needed facts before searching?
Show answer
So you know when to stop and what is still missing. Per-need search answered 17 of 17 correctly, reading 39 pages; one top-3 retrieval answered 14 of 17 and got 13 right.
For the curious Citations need a check
A careless writer, simulated with 25 percent of statements getting a wrong citation or a changed value, produced 4.6 bad citations on average (14.6 of 17 correct) over 30 seeds. A verifier that checks whether the cited page contains the claim, then repairs or regenerates it, gave 0 bad citations and 17 of 17 correct. That check is mechanical on the invented corpus; real pages need text matching or an entailment check and a human spot check.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Plan the facts you need | Then search each one with a budget. |
| Treat pages as untrusted data | Drop instruction-like sentences and mark the page suspect. |
| Rank by source kind, then date, and show disagreement | Report the chosen value and the others. |
| Verify every citation | Does the cited page really say it? |