Part 5 of 11 · 2 min read
What the experiment measured
Coverage and correctness across strategies, verifier impact, and the conflicts actually surfaced.
How to read the first table: all three rows face the same 8 questions and 17 needed facts; "Correct" is the number that matters, and "Pages read" is its cost.
| Strategy (8 questions, 17 needed facts) | Pages read | Answered | Correct |
|---|---|---|---|
| Single retrieval, top 3 pages | 24 | 14 | 13 |
| Per-need search with conflict handling | 39 | 17 | 17 |
| Per-need search, budget of 3 sources per question | 24 | 12 | 12 |
What to notice: the budgeted row reads the same number of pages as the single-retrieval row (24) but gets every one of those 12 answers right. Single retrieval gets 13 right out of a different, larger set of 14. Per-need search spends its page budget more effectively, not just differently.
How to read the verifier table: both rows face the identical careless writer; only whether a verifier runs afterward differs.
| Careless writer (25% wrong), mean of 30 seeds | Bad citations | Correct of 17 |
|---|---|---|
| No verifier | 4.6 | 14.6 |
| Citation verifier plus regenerate | 0.0 | 17.0 |
What to notice: the verifier does not just catch bad citations. Fixing them also recovers the correctness that was lost, because a wrong citation was usually paired with a wrong value.
How to read the conflicts table: each row is one fact where sources disagreed; the chosen value's source kind and date are what won the ranking.
| Conflict surfaced | Chosen (source kind, date) | Disagreeing values |
|---|---|---|
| Zephyr DB p95 latency | 42 ms (primary, 2026-03) | 15 (forum), 80 (blog, 2024) |
| Zephyr DB max nodes | 64 (primary) | 16 (blog, 2024) |
| Lumen Cache license | BSD-3 (primary) | GPL-3 (forum rumour) |
| Orbit Queue p95 latency | 9 ms (primary) | 12 (review, 2025) |
What to notice: every chosen value in this table is from a primary source, so the ranking rule worked as intended on this corpus. That's exactly why the disagreeing values still need to be shown, not discarded. The ranking could be wrong on a corpus with a different mix of sources.
{{svg:pipeline|From question to a verified, cited report.}} {{svg:results|Coverage, correctness and citation checks.}}