Definitions and setup: Caching strategy

Definitions

Prompt cache, prefix, hit and miss

Prompt cache: provider-side storage of the model's intermediate computation for a request prefix.
Prefix: the leading part of a request, taken in order.
Cache hit: the new request's prefix is identical to a stored one; that part is read, not recomputed.
Cache miss: the prefix differs from the first changed token onward; the rest is processed from scratch.

Cache write, cache read, time-to-live (TTL)

Cache write: processing a prefix and storing it. Some providers charge a premium for it, or for storage time.
Cache read: using a stored prefix, billed at a discount.
TTL: how long an entry survives without being used. Most providers keep it short and refresh it on use, which is why steady traffic hits and sparse traffic misses.

Breakpoint, stable prefix, cache-friendly layout

Breakpoint: a marker for where the cacheable part of the request ends (supported explicitly by some providers, automatic in others).
Stable prefix: the leading part that does not change between requests.
Cache-friendly layout: stable content first, volatile content last, history append-only.

The same four requests: green parts are read from the cache, red parts are recomputed. One change near the top spoils the whole request.

Three rules follow from the picture:

Anything that changes every step belongs after everything that does not. A "current time" line in the system prompt turns every request into a full miss.
Order inside the stable part matters too. Tools and the system prompt should be in a fixed order; a reordered tool list is a different prefix.
A hit is by exact content. A one-token edit to a message is a miss from that message onward.