Part 9 of 16 · 1 min read
The prompt-cache trade-off
Editing the middle of a request is not free even when the edit itself looks cheap.
Prompt cache, prefix, cache miss
A prompt cache lets the provider reuse the computation for the leading portion (the prefix) of a request when consecutive requests start with identical content. A change anywhere in the prefix invalidates the cache from that point on, a cache miss, and everything after the change is recomputed.
This is why the strategy is threshold-triggered and why the order matters. Appending to the end of the history keeps the prefix. Editing a tool result in the middle (compaction), or inserting a new summary at the top (summarisation), changes the prefix and makes the next call recompute it. Practical consequences:
- Reduce in batches (a big compaction occasionally), not a little on every step.
- Put the stable parts first (system prompt, tools, state block) and the changing parts last.
- A structured state block that changes rarely is cache-friendlier than a summary that is rewritten every round.
The simulation does not model caching; module 7 does. Treat this section as a cost you must add to the table in Part 5, not as a measured number.