Part 3 of 9 · 2 min read
Recall is lowest in the middle of the input, not at either end
A fact buried in the middle of a long document is read less reliably than the exact same fact at the start or the end.
Think of listening to a long briefing. You remember the opening and the closing clearly, and the middle blurs together. This happens even though every sentence in it was spoken with the same volume and clarity.
In plain words: context rot is accuracy falling as input length grows, lost in the middle is accuracy being lowest for information placed in the middle of the input and highest at the start and end (a U-shaped curve over position), and position bias is the general tendency to weight some positions more than others -- two separate effects that compound, so a long input with information in the middle is hit by both at once.
A worked example. The toy reader's position factor is 1 - dip * 4 * depth * (1 - depth), which equals 1 (no penalty) at depth 0 or 1 (the very start or end) and drops to its lowest point exactly at depth 0.5 (the middle), losing 25% of recall there relative to the ends when dip = 0.25. This position effect multiplies with the separate length effect, so at a long context length both factors are small at once, and the combination is worse than either alone.
The key lines. From code/mini-longctx/reader.py:
# from code/mini-longctx/reader.py
pos = 1 - self.dip * 4 * depth * (1 - depth) # 1 at the ends, 1 - dip in the middle
Read it as: 4 * depth * (1 - depth) is a parabola that peaks at exactly depth = 0.5 and is zero at depth = 0 or depth = 1. This is the U-shape written as one line of arithmetic. The penalty is proportional to how close to the middle the needle sits. It is zero at either end and maximal in the center.
Context rot, lost in the middle, position bias
Context rot is accuracy falling as input length grows. Lost in the middle is accuracy being lowest for information in the middle of the input and highest at the start and end. Position bias is the general tendency of a model to weight some positions more than others; its cause is a research question, not something to state as settled.
Example: a needle at depth 0.5 losing 25% of recall relative to a needle at depth 0 or 1, in a model whose length is otherwise held fixed, is lost in the middle isolated from context rot -- the same length, a different position, a different result.