Part 2 of 11 · 2 min read
Why one retrieval misses what per-need search finds
Planning the facts a question needs, then searching for each one, finds what a single search pass walks right past.
Think of a single trip to a hardware store with a mental list versus a written shopping list checked item by item. Without the list, you come home having forgotten the one thing you actually needed most.
In plain words: a question often needs several separate facts. Searching once for the whole question tends to retrieve pages about the easy, well-covered facts. It tends to miss the ones that needed a more targeted search.
A worked example. Across 8 questions needing 17 facts total, a single top-3 retrieval per question reads 24 pages, answers 14 of 17, and gets 13 right. Planning which facts each question needs first, then searching separately for each one, reads more pages (39) but answers all 17 and gets all 17 right. Capping that per-need search at a 3-source-per-question budget drops it back to 12 of 17 answered — reading more only helps up to the point the budget allows it.
The key lines. From code/mini-research/research.py:
# from code/mini-research/research.py
def iterative(question, index, budget=12, per_need=2):
needs = plan(question)
for need in needs:
e, a = need
for q in (f"{e} {ATTR_WORDS[a]}", f"{e} {ATTR_WORDS[a]} official documentation"):
for i in index.search(q, per_need + 1):
if len(read_ids) >= budget and i not in read_ids:
continue
Read it as: the plan breaks the question into a list of (entity, attribute) needs, and each need gets its own targeted search. A fact that a whole-question search would have missed now gets its own dedicated query. It no longer has to compete for attention with every other fact in the question.
Claim, research budget
A claim is one checkable statement — here a triple of subject, attribute and value; a research budget is the cap on sources read, steps, time or cost for one run.
Example: the 3-sources-per-question budget above, which traded 5 of the 17 facts for staying within a fixed cost.