CSE Roadmap

Core tier · about 8 hours · 16 parts

Part 4 of 16 · 3 min read

Selection is only as good as the signal it ranks by

A selector that finds the answer for a plainly worded question can still fail completely on a question that means the same thing in different words.

Think of a librarian who finds a book only by matching the exact words on its spine to the exact words you say out loud. Ask for "the red book about trains" and they find it instantly. Ask for "the one with the steam engine on the cover" and they come back empty-handed, even though it's the same book.

In plain words: lexical selection ranks items by the words they share with the question rather than by what the question actually means. Presence is a plain, checkable fact about whether the correct answer string reached the window at all. Modelled accuracy is only an assumed probability curve for whether the model would then use it. That is why the two numbers can diverge, and only the first one can be verified by just reading the text.

A worked example. Splitting the same 100 questions by phrasing: at a 1,000-token budget, curated finds the answer for 57% of plainly worded questions ("What is the staging deploy key?") but only 12% of vaguely worded ones ("Which key deploys staging?"), because the plain question shares the words deploy and key with the line that holds the answer while the vague one shares only staging, a word on almost every line. At 4,000 tokens the gap nearly closes (100% versus 85%), but the failure at smaller budgets is real and was not tuned away.

The key lines. From code/mini-window/assemble.py:

# from code/mini-window/assemble.py
def score(text: str, query: str, idf: dict[str, float]) -> float:
    """An item is as relevant as its best line. (Summing over the whole item rewards length: a long log always mentions the environment somewhere.)"""
    return max((line_score(l, query, idf) for l in text.split("\n")), default=0.0)

Read it as: the docstring names the exact design trap this function avoids. Scoring by the best single line rather than by a sum keeps a long, mostly-irrelevant log from outranking a short, directly relevant note. A sum would let the log win just because it has more lines to accumulate points across. The vague-question failure above is a limit of word overlap itself, not of this particular scoring choice.

Presence, modelled accuracy, lexical selection, needle in a haystack

Presence is a plain check on text: is the correct answer string in the context the model would be shown, with no stale value beside it? Modelled accuracy is an assumed probability of the model answering correctly from an invented curve, a direction rather than a measurement. Lexical selection means choosing items by the words they share with the question. A needle in a haystack test hides one fact at different depths of a long input and asks whether it is found.

Example: "12% presence on vague questions at 1,000 tokens" is a fact about the text you can go verify by reading it, while "the modelled score was 0.528" is a number from an assumed curve -- treating the second kind of number as if it were the first is the easiest way to overstate what an experiment like this one actually shows.