Part 12 of 16 · 2 min read
Answering context-window questions in an interview
Why not just put everything in a very long context?
"Four honest reasons, and I can show each one in a toy experiment: it gets refused past the window limit; it costs more at every step, since the total grows with the square of the number of steps; corrections sit beside old values, so the answer is present and so is the wrong one; and long inputs are read less reliably regardless. The fix is to keep a store outside the window and build each window from it under a budget." Follow-up: "How much more does it cost?" In a 60-step run, keeping everything sent 4.5 times more tokens than a curated budget of 2,000, and it was refused midway through while the curated run never was.
What goes in the context window, and who decides?
"At every step the agent builds the context from a larger store under a token budget. Rules are pinned. Corrected facts replace the old versions. Long tool results are trimmed to the relevant lines. The rest is ranked by relevance and packed to the budget, the question goes last, and everything from outside is escaped so it cannot forge structure." Follow-up: "How do you know it worked?" I check three things separately: the answer was in the window, the budget was respected, and cost per step stayed flat.
How do you stop a tool result from pretending to be a system instruction?
"Wrap every item in a header the assembler writes itself, and escape that header's special characters inside every item's own text before wrapping it, so nothing inside an item can ever form a second, fake header." Follow-up: "Does that stop prompt injection?" No -- it stops a forged structural block, not a persuasive sentence in plain English. Stopping the model from being talked into something still needs least privilege and approval outside the model, which is why fencing is hygiene, not a complete defence.