Part 7 of 11 · 2 min read
Answering caching-strategy questions in an interview
How do you reduce the cost of an agent that re-sends a long history every step?
"I use the provider's prompt cache and I shape the request for it: stable content first (system prompt, tools, standing state), history append-only, and the new turn last, so each step reads the old prefix from the cache and pays full price only for what is new. I keep anything that changes per step, like a timestamp, out of the prefix. In the example's simulation that took the cost of a 40-step run to 15 percent of the uncached cost, and a timestamp in the system prompt made it 125 percent." Follow-up: "What if the history gets too long?" I reduce it, but in batches: compaction edits old messages, which misses the cache from the edit onward, so compacting every step cost twice as much as leaving the history alone even though the window was smaller.
Does compaction save money or cost money?
"It depends on the cache discount, and I'd measure it. Compaction shrinks every later request but breaks the prefix at the edit. With a 90 percent read discount, compacting every step was more expensive than append-only; with a 50 percent discount it was cheaper. Compacting in batches was cheapest at every price I tried, because it keeps most steps append-only." Follow-up: "Where would you set the threshold?" Where the cost of a long window plus the accuracy loss from length (module 6) exceeds the cost of one miss; I'd compute it from my own prices and traffic.
Why did the cache stop working?
"I'd look in this order: something volatile near the top (a timestamp, an id, a reordered tool list); an edit to an earlier message; a gap longer than the TTL; a prefix below the minimum cacheable size; or a change to the model, tool definitions or system prompt, which is a different prefix. Then I'd check the provider's cached-token count per call, which tells me directly." Follow-up: "Is a cache hit a correctness risk?" No: it reuses computation for identical input, so the output distribution is the same; the risks are cost and latency, and privacy if a cache were shared across tenants, which is a provider question.