CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 8 of 11 · 2 min read

Mistakes that silently break the cache

Common mistakes, and what the example does not do.

Mistakes that silently break the cache

  • A timestamp, request id or user name in the system prompt.
  • Reordering tools or sections between requests.
  • Compacting or summarising on every step.
  • Rewriting the notes or state block every step.
  • Quoting one provider's discount as universal. Terms differ and change.
  • Ignoring idle time in low-traffic agents.

What the example does not do

About this example What the example does not do
  • No real provider. Real caches have minimum sizes, breakpoints, per-provider quirks and storage fees; the example has a minimum size, a TTL and two multipliers.
  • Abstract content. Segments are identities and token counts, so tokenisation effects (a small edit changing token boundaries) are not modelled.
  • One agent shape. Parallel sub-agents sharing a prefix, or a shared cache across users, behave differently (module 8 and your provider's documentation).
  • Optional real-provider experiment: make the same 40 steps with a real API and compare the reported cached-token counts for append-only and compact-every-step.

Check yourself

You add the current time to the first line of the system prompt so the model knows the date. What happens to your cache hits?

Show answer

They drop to zero. The cache needs an identical prefix, and the first changed token ends the hit. Put the time in the newest message instead.

For the curious Compaction can cost more than it saves

Compacting every step shrank the window by 43 percent in the example but cost almost twice as much as leaving the history alone, because every rewrite broke the cached prefix. Compacting in batches was cheapest. If the cache read price is high (0.5 of normal), the answer flips, so the result depends on your provider price.

What you will use daily at work

Practice What it looks like
Stable first, volatile last System prompt, tools, notes, history, then the new turn.
Watch idle time An entry that expires between requests turns the next request into a full write.
Read your provider cache fields Most responses report cached and uncached tokens separately.