Part 8 of 11 · 2 min read
Mistakes that silently break the cache
Common mistakes, and what the example does not do.
Mistakes that silently break the cache
- A timestamp, request id or user name in the system prompt.
- Reordering tools or sections between requests.
- Compacting or summarising on every step.
- Rewriting the notes or state block every step.
- Quoting one provider's discount as universal. Terms differ and change.
- Ignoring idle time in low-traffic agents.
What the example does not do
About this example What the example does not do
- No real provider. Real caches have minimum sizes, breakpoints, per-provider quirks and storage fees; the example has a minimum size, a TTL and two multipliers.
- Abstract content. Segments are identities and token counts, so tokenisation effects (a small edit changing token boundaries) are not modelled.
- One agent shape. Parallel sub-agents sharing a prefix, or a shared cache across users, behave differently (module 8 and your provider's documentation).
- Optional real-provider experiment: make the same 40 steps with a real API and compare the reported cached-token counts for append-only and compact-every-step.
Check yourself
You add the current time to the first line of the system prompt so the model knows the date. What happens to your cache hits?
Show answer
They drop to zero. The cache needs an identical prefix, and the first changed token ends the hit. Put the time in the newest message instead.
For the curious Compaction can cost more than it saves
Compacting every step shrank the window by 43 percent in the example but cost almost twice as much as leaving the history alone, because every rewrite broke the cached prefix. Compacting in batches was cheapest. If the cache read price is high (0.5 of normal), the answer flips, so the result depends on your provider price.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Stable first, volatile last | System prompt, tools, notes, history, then the new turn. |
| Watch idle time | An entry that expires between requests turns the next request into a full write. |
| Read your provider cache fields | Most responses report cached and uncached tokens separately. |