Part 7 of 9 · 1 min read
Quick notes
- Context rot is the degradation of a model's reliability as the input gets longer, well below the advertised window. The effective context length is the largest input at which the model still meets your accuracy target, and it is the smaller number.
- Needle in a haystack is the standard probe, and a clean one is the easy case: one needle, no near-duplicates, a question that matches the needle's words.
- Mitigations, in order of preference: put less in the window, put the right things in, place critical information at the start or end, repeat the key instruction near the end, remove near-duplicates.
- Set your budget below your measured effective length, not the advertised one.
- The example's reader is a toy with an assumed curve, so its heatmap shows how to read a map, not how any real model behaves.