CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 8 of 11 · 2 min read

Mistakes that produce a fluent, wrong report

Common mistakes, and what the example does not do.

Mistakes that produce a fluent, wrong report

  • One retrieval for a multi-part question, then confident synthesis.
  • Citing a page that does not contain the claim.
  • Hiding conflicts to make the answer look clean.
  • Trusting forum posts or very old pages without dates.
  • Treating page text as instructions.
  • Counting repeated claims as independent confirmation.
  • No "not found" path, so the agent invents.
  • No budget, so one hard question burns the day's spend.
  • Evaluating only the prose, not the claims and sources.
  • Dropping source dates and URLs from the report.

What the example does not do

About this example What the example does not do
  • No model and no web: the planner is rules, the writer is a script and the corpus is invented.
  • Claims are structured (CLAIM entity | attribute | value), which makes verification exact; real pages need extraction, matching and judgement, which are the hard part.
  • The corpus is tiny, so retrieval is easy; real corpora have duplicates, near-duplicates, paywalls and noise.
  • The injection defence is a regular expression, easy to evade; it demonstrates the structure (data, not instructions), not a robust filter.
  • No citation-laundering detection, no source-origin tracing, no handling of numbers with units or ranges.
  • The ranking rule (kind, then date) is a simple choice; recency can be wrong when an old primary source is the origin.
  • Optional real experiment: pick ten factual questions with known answers in a domain you know, run a real model with search, and score coverage, correctness, citation precision (open each cited page) and cost. Then add a verifier and repeat.

Check yourself

Why plan the needed facts before searching?

Show answer

So you know when to stop and what is still missing. Per-need search answered 17 of 17 correctly, reading 39 pages; one top-3 retrieval answered 14 of 17 and got 13 right.

For the curious Citations need a check

A careless writer, simulated with 25 percent of statements getting a wrong citation or a changed value, produced 4.6 bad citations on average (14.6 of 17 correct) over 30 seeds. A verifier that checks whether the cited page contains the claim, then repairs or regenerates it, gave 0 bad citations and 17 of 17 correct. That check is mechanical on the invented corpus; real pages need text matching or an entailment check and a human spot check.

What you will use daily at work

Practice What it looks like
Plan the facts you need Then search each one with a budget.
Treat pages as untrusted data Drop instruction-like sentences and mark the page suspect.
Rank by source kind, then date, and show disagreement Report the chosen value and the others.
Verify every citation Does the cited page really say it?