CSE Roadmap

Core tier · about 8 hours · 16 parts

Part 6 of 16 · 2 min read

What the example found

Presence by policy and budget, where lexical selection breaks, the position effect, and the cost of keeping everything.

How to read the presence-by-budget table: each row is a policy; each column is a token budget; the value is the share of 100 questions whose correct answer reached the window with no stale value beside it.

Policy 500 1,000 2,000 4,000 8,000
recent 20% 21% 26% 32% 53%
relevant 33% 32% 42% 57% 71%
curated 35% 48% 68% 94% 100%

What to notice: curated reaches 94% at 4,000 tokens, while relevant alone needs 8,000 tokens to reach just 71%. Supersede and trim are what let a small budget be enough, not a smarter ranking of what to keep.

How to read the lexical-selector table: each row is a token budget for curated; the columns split the same 100 questions by how plainly they were worded.

curated, budget plain words vague words the rule
1,000 57% 12% 100%
2,000 82% 38% 100%
4,000 100% 85% 100%

What to notice: the rule column sits at 100% at every budget because it is pinned. The vague column lags the plain column by as much as 47 points at the same budget. That gap is the honest cost of ranking by shared words rather than by meaning.

How to read the cost table: each row compares total tokens sent across all 60 steps of one run under two policies.

Policy over 60 steps Total tokens sent Refused from
keep_all 482,308 step 46
curated at budget 2,000 106,539 never

What to notice: keep_all sends 4.5 times more tokens than curated and still gets refused partway through the run. A bigger window does not fix this, because the total grows with the square of the number of steps, while a fixed budget grows in a straight line.

{{svg:budget|The answer's presence in the window, by policy and budget. The two red lines are the keep-everything policy under two assumptions about the window.}} {{svg:growth|Tokens sent at each of 60 steps. The dashed line is the imagined limit.}}