CSE Roadmap

Core tier · about 5 hours · 9 parts

Part 2 of 9 · 3 min read

The window a model accepts is not the window it reads well

A model that accepts 200,000 tokens and a model that stays accurate for 200,000 tokens are two different claims.

Think of a car's fuel tank rated at 60 litres. That is how much it holds. It is not how far the car drives efficiently on a full tank. The two numbers measure different things. Only one of them tells you what to plan a trip around.

In plain words: the context window is the maximum number of tokens a model accepts in one call, the advertised context is that number on the model card, and the effective context length is the longest input at which the model still meets a stated accuracy threshold on your task -- a smaller number than the advertised one, and one that only means something stated alongside the task and the threshold it was measured against.

A worked example. On the toy reader (whose chance of finding a needle falls as 1 / (1 + (length / 24,000)^2)), a 90% recall threshold is never met at any length tested, a 70% threshold holds up to 4,000 tokens, and a 50% threshold holds up to 8,000 tokens. Three different thresholds on the exact same model produce three different "effective lengths" -- which is why quoting a single number with no threshold attached is not yet a complete claim.

The key lines. From code/mini-longctx/reader.py:

# from code/mini-longctx/reader.py
l0: float = 24000.0       # context length (tokens) at which the length factor halves

def p(self, length: int, depth: float, distractors: int = 0, copies_at_ends: bool = False) -> float:
    pos = 1 - self.dip * 4 * depth * (1 - depth)               # 1 at the ends, 1 - dip in the middle
    ln = 1 / (1 + (length / self.l0) ** 2)
    return min(1.0, max(self.floor, pos * ln * (1 - self.distractor_loss) ** distractors))

Read it as: ln is one multiplicative factor among several, and it depends only on length, never on where the needle sits. Length and position are modeled as two separate effects that both reduce recall. That is why a single "effective length" number has to be read off a map, not a curve. The position still matters once you have fixed the length.

Context window, advertised context, effective context length

The context window is the maximum number of tokens (input plus output) a model accepts in one call. The advertised context is that number on the model card. The effective context length is the longest input at which the model still meets a stated accuracy threshold on your task; it depends on the task, the threshold and where the information sits, so it is always given with all three.

Example: "90 percent is never met, 70 percent holds to 4,000 tokens, 50 percent to 8,000" is effective context length stated correctly -- three numbers, each tied to its own threshold, rather than one number standing in for all of them.