CSE Roadmap

Core tier · about 5 hours · 9 parts

Part 6 of 9 · 3 min read

Answering context-rot questions in an interview

Questions with follow-ups, common mistakes, and limits.

Answering context-rot questions in an interview

The model supports a one-million-token window. Why not put everything in it?

"Because the window is an upper bound on what the model accepts, not on what it uses well. Reported evidence says accuracy degrades as input grows, often well below the limit, and information in the middle is used less reliably than at the start or the end. Cost and latency also grow with length. So I'd measure my own effective context length with a needle test on my own data, with distractors, and keep the working context below that, using compaction, offloading and retrieval to get there." Follow-up: "How do you measure it?" Insert a known fact at several depths in inputs of several lengths, ask for it, and map recall by length and depth; the effective length is where the worst depth falls below the target I set. Follow-up: "What's wrong with that test?" A plain needle is the easy case: one match, no near-duplicates. I add distractors and paraphrases, and I use private data so the model can't answer from training.

Where do you put the important instructions in a long prompt?

"At the start, for the standing rules, and repeated or restated at the end, just before the question, for the thing that must not be missed. The middle is the weakest place in the reported results.

Context rot or lost in the middle: what's the difference?

"Context rot is the overall decline with length. Lost in the middle is a position effect within a given length: start and end are better than the middle. They compound: at a long length the whole curve is lower and the middle is lowest. A recall map over length and depth shows both."

Mistakes that overstate what a model can read

  • Treating the advertised window as the usable one.
  • Reporting an effective length with no threshold or task.
  • Testing only a clean needle. No distractors, no paraphrase, no second fact.
  • Placing the critical instruction in the middle.
  • Filling the window "because it fits". Cost, latency and accuracy all move against you.
  • Quoting a toy result as model evidence. This module's numbers are for learning the method.

What the example does not do

About this example What the example does not do
  • No real model. The recall numbers are properties of the toy curve.
  • One needle, one question type. No multi-hop, aggregation or reasoning over the whole context.
  • Filler is repetitive and easy. Real documents have structure, topic drift and genuine near-duplicates.
  • No output-length effects, no cost or latency measurement, and no comparison across models.
  • Optional real-model experiment: implement reader.answer(context, question) with a call to your model and run harness.grid. Report the map, the effective length at 90 and 70 percent, and the effect of five distractors.

Check yourself

Your model accepts 200,000 tokens. Does that mean you can send 200,000 and expect good answers?

Show answer

No. The advertised window is what it accepts; the effective length is how long it stays accurate on your task, and that is shorter. Measure it.

For the curious A clean needle test is the easy case

The probe puts one fact in filler text and asks for it in the same words. Real work has similar passages nearby, paraphrased facts and questions that need several facts. On the toy reader here recall fell from 59 percent with no distractors to 19 percent with ten, and that size is an assumption; the direction is the reason to test with them.

What you will use daily at work

Practice What it looks like
A recall map Length on one side, depth on the other, recall in each cell.
A budget below the measured length Set the limit from your test, not from the advertised window.
Retrieve before you read Send the relevant few thousand tokens, not the whole store.