Part 6 of 11 · 1 min read
Build it so the cache holds up
A practical layout, and the rules behind it.
[ system prompt ] stable for weeks
[ tool definitions ] stable per release, fixed order
[ standing notes / state ] changes rarely; update in batches
[ history ... ] append-only
[ new user turn ] the only uncached part
- Put per-step values in the last segment, not the system prompt (time, user id, request id).
- Do not rewrite the state block every step; change it when it must change, and expect one miss from that point.
- Compact on a threshold and to a target well below it, so the next compaction is many steps away.
- Keep the traffic warm if the cost of a miss is high and the TTL is short, or accept the miss for sporadic work.
- Watch the minimum size. Below the provider's minimum cacheable length, nothing is cached.
- Measure. Providers report cached and uncached input tokens per call; chart the read share per step (tokens per step is the number to watch).