Part 6 of 7 · 3 min read
The costs of a cache
What is paid for the speed: stale data, memory, and a second system.
The costs of a cache
Think of the notes a student keeps on the desk. They save a trip to the library, but if the book is revised, the notes are wrong until they are rewritten. The student must also decide which notes to keep when the desk is full.
The book "System Design Interview" lists the considerations in using a cache. They map onto costs:
- Staleness (consistency). The data store and the cache are not changed in one transaction, so they can disagree. Keeping them in sync is hard, and harder across regions. The book points to a paper about Facebook's memcache for the details.
- Memory. Cached data takes memory, which is dearer than disk. The cache has a limit, so something must be removed when it is full (module 3).
- Expiry. Without an expiry policy, data stays forever. Too short a lifetime sends too many reads to the database. Too long makes the data stale.
- A new point of failure. One cache server is a single point of failure. The book recommends several cache servers across data centres, and spare memory.
- Volatility. Memory is lost on restart, so the cache is never the only copy.
- Complexity. More code paths, more things to monitor, harder bugs.
A worked example of the staleness cost. A price is cached with a time to live of 60 seconds. The price changes at second 10. Readers see the old price from second 10 to second 60, which is 50 seconds. If 200 pages are viewed per second, then 50 × 200 = 10,000 views show the old price. A shorter lifetime of 10 seconds cuts that to at most 10 seconds of staleness, but raises database reads by up to 6 times for that key (one refresh per 10 seconds, not per 60).
The book "Building Microservices" adds advice that holds well: keep caching simple. The more caches between the reader and the source, the staler the data can be, and the harder it is to say how fresh a value is. It suggests sticking to one cache where one will do, and thinking before adding more.
A cache can also help when things go wrong. The same book notes that a cache can serve stale data when the source is down, which for some systems is better than returning an error. That is a business decision about how stale is acceptable.
Interview angle
Answer in your head first, then open it.
Why add a cache?
Reads are many, changes are few, and a miss is costly. State the numbers: hit ratio, the load left on the database, and the saving per request.
What does the cache cost?
Staleness, memory, expiry choices, a new failure point and more code. Name how stale a value may be and why that is fine.
How big should it be?
Estimate the hot set, not the whole data. Show the arithmetic, as in the 20 GB and 4 GB example.
Is the cache the source of truth?
No. It holds copies that can be rebuilt from the database.
Check yourself:
- Name two costs of a cache that a database alone does not have.
- A price cache has a 30-second TTL and a price changes. What is the longest time a reader can see the old price?
Words you will meet
| Word | Plain meaning |
|---|---|
| Cache | A small, fast store of copies from a slower source |
| Hit | The item was found in the cache |
| Miss | The item was not in the cache and must be fetched |
| Hit ratio | Hits divided by requests |
| Locality of reference | Requests tend to repeat or cluster on a few items |
| Skew | A small share of the items gets most of the requests |
| Hot set | The busy items that most requests ask for |
| Latency | The time one operation takes |
| Stale data | A cached copy that no longer matches the source |
| TTL | Time to live: how long an item may stay in the cache |
| Best-effort replication | Copies that are kept but not guaranteed to be current or present |