Part 6 of 16 · 2 min read
What the example found
Presence by policy and budget, where lexical selection breaks, the position effect, and the cost of keeping everything.
How to read the presence-by-budget table: each row is a policy; each column is a token budget; the value is the share of 100 questions whose correct answer reached the window with no stale value beside it.
| Policy | 500 | 1,000 | 2,000 | 4,000 | 8,000 |
|---|---|---|---|---|---|
recent |
20% | 21% | 26% | 32% | 53% |
relevant |
33% | 32% | 42% | 57% | 71% |
curated |
35% | 48% | 68% | 94% | 100% |
What to notice: curated reaches 94% at 4,000 tokens, while relevant alone needs 8,000 tokens to reach just 71%. Supersede and trim are what let a small budget be enough, not a smarter ranking of what to keep.
How to read the lexical-selector table: each row is a token budget for curated; the columns split the same 100 questions by how plainly they were worded.
curated, budget |
plain words | vague words | the rule |
|---|---|---|---|
| 1,000 | 57% | 12% | 100% |
| 2,000 | 82% | 38% | 100% |
| 4,000 | 100% | 85% | 100% |
What to notice: the rule column sits at 100% at every budget because it is pinned. The vague column lags the plain column by as much as 47 points at the same budget. That gap is the honest cost of ranking by shared words rather than by meaning.
How to read the cost table: each row compares total tokens sent across all 60 steps of one run under two policies.
| Policy over 60 steps | Total tokens sent | Refused from |
|---|---|---|
keep_all |
482,308 | step 46 |
curated at budget 2,000 |
106,539 | never |
What to notice: keep_all sends 4.5 times more tokens than curated and still gets refused partway through the run. A bigger window does not fix this, because the total grows with the square of the number of steps, while a fixed budget grows in a straight line.
{{svg:budget|The answer's presence in the window, by policy and budget. The two red lines are the keep-everything policy under two assumptions about the window.}} {{svg:growth|Tokens sent at each of 60 steps. The dashed line is the imagined limit.}}