Part 9 of 11 · 1 min read
Quick notes
- A hit needs an identical prefix. The first changed token ends the hit; everything after it is recomputed, and on most providers written to the cache again at a premium.
- Therefore order the request from most stable to most volatile: system prompt, tool definitions, standing notes, history, then the new turn. Never put a timestamp, a request id or any per-step value near the top.
- Quote these numbers, with the caveat that the prices are the parameters (read at 0.1 of the normal input price, write at 1.25, time-to-live of 5 steps): over 40 steps, append-only with caching cost 0.15 of no caching; a timestamp in the header cost 1.25, worse than no cache; compacting every step cost 0.29, almost twice append-only even though the window was 43 percent smaller; compacting in batches cost 0.14, the cheapest.
- The trade-off flips with the discount: at a read price of 0.5, compacting every step (0.44) beats append-only (0.54). So "compaction saves money" or "compaction costs money" are both wrong without the price.
- Idle time matters: an entry that expires between requests turns the next request into a full-price write.