CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 4 of 11 · 2 min read

Order the request from what never changes to what always does

Where a piece of the prompt sits is itself a design decision, not just what the piece contains.

Think of packing a suitcase for a trip with several stops. The clothes you will not touch until the last day go at the bottom. Whatever you need right away goes on top. Getting that order backwards means repacking the whole bag every single day.

In plain words: a breakpoint marks where the cacheable part of a request ends, a stable prefix is the leading part that never changes between requests, and a cache-friendly layout puts stable content first and volatile content last with history kept append-only -- because the cache can only ever help the part of the request that comes before the first thing that changed.

A worked example. The practical layout places the system prompt and tool definitions first (stable for weeks or a release), standing state next (changes rarely, updated in batches), then append-only history, with the new user turn last as the only segment that's expected to differ every time. Compacting the history every single step, instead of in batches, edits an old message and breaks the cached prefix at that edit point -- this alone raises the 40-step run's relative cost to 0.29, almost twice the 0.15 of simply leaving the history alone, even though the window itself shrank by 43%.

The key lines. From code/mini-cache/cache.py:

# from code/mini-cache/cache.py
def send(self, request: list[Segment], step: int, cache_upto: int | None = None) -> Result:
    """Send a request. Segments [0, cache_upto) are marked cacheable (default: all but the last segment, the new turn)."""
    req = tuple(request)
    k = len(req) - 1 if cache_upto is None else cache_upto

Read it as: the default breakpoint k is "every segment except the last one." The function's own default encodes the layout rule directly. Whatever is in the final segment is assumed to be the one thing that changes every request. Everything before it is assumed stable enough to be worth caching.

Breakpoint, stable prefix, cache-friendly layout

A breakpoint marks where the cacheable part of the request ends. A stable prefix is the leading part that does not change between requests. A cache-friendly layout puts stable content first, volatile content last, with history kept append-only.

Example: batch compaction (editing the history rarely, to well below the budget) costing 0.14 -- cheaper than never compacting at all -- against every-step compaction's 0.29 is the breakpoint-and-layout idea paying off exactly where it was designed to: most steps stay append-only, so most steps stay cache-friendly.