Part 3 of 11 · 2 min read
A read is cheap, a write costs extra, and an idle entry expires
A cache that nobody reuses in time has already cost more than never caching at all.
Think of a seat held at a restaurant. The host only holds it for so long. If nobody shows up in time, the table goes to someone else. The next party to want that spot pays full price to be seated again, as if the hold never happened.
In plain words: a cache write is processing a prefix and storing it, usually billed at a premium over an ordinary token; a cache read is using a stored prefix at a discount; and a time-to-live (TTL) is how long an entry survives without being reused -- refreshed on every hit, which is why steady traffic keeps hitting the cache and sparse traffic keeps missing it.
A worked example. With a cached-read price of 0.1 (90% discount) and a write premium of 1.25, an idle gap longer than the TTL inserted before every fifth step raises the 40-step run's relative cost from 0.15 to 0.39 -- because each gap forces a fresh write at the premium price instead of a cheap read. Widening the TTL to 15 steps (long enough that the same gap no longer exceeds it) brings the cost straight back down to 0.15: the entries never expire between uses, so every step after the first is a read again.
The key lines. From code/mini-cache/cache.py:
# from code/mini-cache/cache.py
ttl: int = 5 # steps an entry survives without being used (refreshed on use)
...
if write_tokens or hit:
self.entries = [(sq, l) for sq, l in self.entries if step - l <= p.ttl] # drop expired entries
self.entries.append((req[:k], step)) # the cached prefix of this request is live from now, so a hit refreshes it
Read it as: every use, whether a write or a hit, re-appends the entry with the current step as its new last_used. This is what "refreshed on use" means in code. The clock restarts on every request that touches the cache. So steady traffic never lets an entry go idle long enough to expire. Traffic with a gap longer than the TTL always finds it gone.
Cache write, cache read, time-to-live (TTL)
A cache write is processing a prefix and storing it; some providers charge a premium for it. A cache read is using a stored prefix, billed at a discount. TTL is how long an entry survives without being used; most providers keep it short and refresh it on use.
Example: the cost curve going 0.15 -> 0.39 -> 0.15 as the gap-versus-TTL relationship flips twice is TTL expiry made visible as three numbers on the same 40-step run -- nothing else in the setup changed between them.