CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 2 of 11 · 2 min read

A cache hit needs an identical prefix, not just a similar one

One changed token at the top of a request turns everything after it into full-price work.

Think of photocopying a shared stack of pages. The copies only save time if the stack is byte-for-byte the same as last time. Change the first page and every page after it has to be reprinted, even though most of the stack never changed.

In plain words: a prompt cache is provider-side storage of the model's intermediate computation for a request prefix, and a cache hit happens only when the new request's leading segments are exactly identical to a stored prefix -- the moment one segment differs, that is a cache miss from the first changed token onward, and everything after it is processed from scratch, however small the original edit was.

A worked example. Across a 40-step agent run, appending only new content to a stable prefix (system prompt, tools, history) keeps 95% of all tokens read from the cache, for a relative cost of 0.15 against an uncached run. Adding one volatile line -- a timestamp -- to the top of the system prompt drops the cache-read share to 0%: every single request now differs from the first segment onward, so nothing is ever a hit, and the run costs 1.25 times an uncached run, because every request also pays a write premium it never recovers.

The key lines. From code/mini-cache/cache.py:

# from code/mini-cache/cache.py
def _hit(self, req: tuple, step: int) -> int:
    """Segments of `req` that are a prefix of some live entry."""
    best = 0
    for seq, last in self.entries:
        if step - last > self.pricing.ttl:
            continue
        n = 0
        for a, b in zip(seq, req):
            if a != b:
                break
            n += 1
        best = max(best, n)
    return best

Read it as: the inner loop breaks the instant a != b, so n stops counting at the very first mismatched segment. There is no partial credit for "mostly the same." The hit length is exactly how many segments survived unchanged from the start. That is why a single volatile line at the top can zero out the hit for an entire multi-thousand-token request.

Prompt cache, prefix, cache hit, cache miss

A prompt cache is provider-side storage of the model's intermediate computation for a request prefix. A prefix is the leading part of a request, taken in order. A cache hit means the new request's prefix is identical to a stored one, so that part is read, not recomputed. A cache miss means the prefix differs from the first changed token onward.

Example: the timestamp experiment going from 0.15 to 1.25 relative cost, purely from one line moving to the top of the request, is cache-hit-or-miss stated as money -- there is no partial hit, so there is no partial saving either.