CSE Roadmap

Core tier · about 5 hours · 9 parts

Part 4 of 9 · 2 min read

A needle test maps recall against length and depth, with distractors as the realism check

A plain needle test without near-miss distractors is the easiest possible version of the problem a real document actually poses.

Think of a fire drill with one clearly-marked exit and no smoke: it tells you the door works, but nothing about whether people can find it under real conditions.

In plain words: a needle is a short fact inserted into filler text (the haystack) at a chosen depth, a distractor is a near-miss that looks like the needle but is wrong, and recall is the share of trials in which the model returns the needle's value -- so running the test with distractors added is what turns an easy, unrealistic probe into one that resembles how a real document, with several similar-looking passages, actually behaves.

A worked example. On the toy reader, at a fixed 8,000-token length with the needle in the middle, recall is 59% with zero distractors, 46% with two, 34% with five, and 19% with ten. Each added near-miss ("The access code for the garage is 7310") costs a further 15% of the remaining recall, by design -- the direction of this effect (distractors hurt) is the thing worth testing for on a real model; the exact size is this module's own assumption.

The key lines. From code/mini-longctx/reader.py:

# from code/mini-longctx/reader.py
d = len(re.findall(r"access code for the (?!vault)", context))
p = self.p(length, depth, d)

Read it as: d counts every occurrence of the needle's phrasing that is NOT about the vault. A negative lookahead picks out exactly the near-misses. That count is what the distractor_loss term in p() multiplies against. The test's realism comes entirely from how many of these look-alikes are planted, not from the needle itself.

Needle, haystack, depth, distractor, recall

A needle is a short fact inserted into the input. The haystack is the filler text around it. Depth is where the needle sits, as a fraction of the input (0 = start, 1 = end). A distractor is a near-miss that looks like the needle but is wrong. Recall is the share of trials in which the model returns the needle's value.

Example: recall falling from 59% to 19% as distractors rise from zero to ten, at the exact same length and depth, isolates the distractor effect from both the length and position effects covered in Parts 1 and 2 -- three separate knobs, each shown to matter on its own.