CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 6 of 11 · 1 min read

Build it so the cache holds up

A practical layout, and the rules behind it.

[ system prompt ]            stable for weeks
[ tool definitions ]         stable per release, fixed order
[ standing notes / state ]   changes rarely; update in batches
[ history ... ]              append-only
[ new user turn ]            the only uncached part
  • Put per-step values in the last segment, not the system prompt (time, user id, request id).
  • Do not rewrite the state block every step; change it when it must change, and expect one miss from that point.
  • Compact on a threshold and to a target well below it, so the next compaction is many steps away.
  • Keep the traffic warm if the cost of a miss is high and the TTL is short, or accept the miss for sporadic work.
  • Watch the minimum size. Below the provider's minimum cacheable length, nothing is cached.
  • Measure. Providers report cached and uncached input tokens per call; chart the read share per step (tokens per step is the number to watch).