Part 7 of 8 · 4 min read
Cache poisoning, and the checklist
A wrong value that stays for ever, and one table of all the problems.
Cache poisoning, and the checklist
Think of a sign that says the shop is closed, hung on the door by mistake and never taken down. Customers read it and leave. The shop could be open, but nobody asks.
The book "Building Microservices" tells a cautionary story it calls cache poisoning. A team ran a proxy in front of a legacy application and set proper cache headers on its responses. A bug in the header code let a few pages through unchanged. The legacy application had earlier been changed to send an "Expires: Never" header, which had done no harm while the proxy overrode it. Now it reached users. The team fixed the code and cleared the proxy cache, but that was not enough: the pages were also held in the browsers of many users, where nobody could clear them. The only remedy was to change the URLs of those pages so that they were fetched again. The lesson is that a cache can keep a mistake alive for a long time, in places the team does not control.
Cache poisoning
Cache poisoning is the storing of a wrong or harmful value in a cache, so that readers keep getting it until the entry is removed.
Example: a bug makes a page return "price: 0". A cache stores that response for 24 hours. Every customer sees a price of 0 all day.
Causes and guards:
- Caching error responses. Store only successful answers, or give errors a very short TTL (a few seconds).
- Caching responses built from partial data. If a service times out and returns an empty list, do not store the empty list for long.
- Wrong cache keys. If the key leaves out something that changes the answer (language, user, permissions), one user's page can be served to another. Include everything that changes the response (Level 1, module 4).
- No way to purge. Keep a tested way to delete one key, a prefix or the whole cache.
- Long TTLs on important data. A long TTL makes a mistake last longer.
The checklist of the module:
| Problem | Trigger | Main fix |
|---|---|---|
| Stampede | A hot entry expires | Coalescing, early refresh, stale while refreshing |
| Penetration | Keys that do not exist | Cache "not found" briefly, Bloom filter, validate, rate limit |
| Avalanche | Many entries expire or the cache dies | Jitter, several nodes, shed load, warm up |
| Hot key | One key gets too much traffic | Local copy, key splitting |
| Stale data | Source changes, or copies differ | TTL, invalidation, versioned keys, read-your-writes |
| Poisoning | A wrong value is stored | Do not cache errors, careful keys, ability to purge |
Interview angle
Answer in your head first, then open it.
What happens when a popular cache entry expires?
A stampede: many requests miss at once. Fix it with request coalescing or a lock, and refresh before expiry. Give the numbers.
What if someone asks for keys that do not exist?
Penetration. Cache a short "not found" marker, use a Bloom filter in front, and validate input.
What if the cache dies?
Avalanche. Several nodes, load shedding, warm-up, and an origin sized for the load that gets through.
How do you keep the cache and database consistent?
They cannot be changed in one transaction. Bound the gap with a TTL, invalidate after writes, and read the database where staleness is not acceptable.
One key gets most of the traffic. What do you do?
A hot key. Keep a short-lived local copy or split the key into copies.
Check yourself:
- Why should an error response be cached for only a few seconds, if at all?
- Which problem is solved by a Bloom filter, and which by jitter?
Words you will meet
| Word | Plain meaning |
|---|---|
| Stampede, thundering herd | Many requests miss at once and all hit the database |
| Request coalescing | Only one request loads a missing key; the others wait |
| Penetration | Requests for keys that exist nowhere, so they always miss |
| Negative caching | Storing a "not found" answer briefly |
| Bloom filter | A small structure that says "definitely absent" or "maybe present" |
| Avalanche | A large part of the cache fails at once |
| Jitter | A random extra time on each TTL |
| Hot key | A key that gets more traffic than one server can carry |
| Key splitting | Storing a value under several keys on different servers |
| Read-your-writes | A user sees their own changes at once |
| Versioned key | A key that includes a version number |
| Cache poisoning | A wrong value stored and served for a long time |
| Load shedding | Rejecting some requests at once to protect a limited resource |