Part 5 of 7 · 2 min read
What is worth caching
The tests that tell a good candidate from a bad one.
Think of a restaurant deciding what to prepare in advance. Soup that many guests order can be made in a big pot. A dish that is ordered once a month is cooked to order. The kitchen prepares what is common.
The book "System Design Interview" gives the rule for when to use a cache: when data is read often but changed rarely. It adds that cached data lives in volatile memory, so a cache server is not a place to keep data that must survive. Important data must be saved in a persistent store.
Four tests help to judge a candidate:
- Read-to-write ratio. Does the item get read many times for each change? A product description read 5,000 times a day and changed once a month is excellent. A bank balance changed on every payment is poor.
- Cost of a miss. How slow or costly is the source? A query that joins five tables is a better candidate than a lookup by primary key.
- Tolerance for stale data. Can users see a version that is a few seconds or minutes old? Cached prices that are a minute old may be fine. Cached stock counts at the moment of payment are not.
- Size of the hot set. Does the busy part of the data fit in memory?
A worked example of the last test. A site has 10,000,000 items of 2 KB each. Total size is 10,000,000 × 2 KB = 20 GB. With the skew from section 1 (20 percent of items serve 80 percent of reads, made up), the hot set is 2,000,000 items × 2 KB = 4 GB. A cache of 4 GB serves about 80 percent of the reads. Doubling it to 8 GB might lift the share to 90 percent (made up), but each further doubling gains less. The shape is steep at first and flat later, so size the cache at the point where the gain per gigabyte becomes small.
| Candidate | Reads per change | Miss cost | Stale ok? | Verdict |
|---|---|---|---|---|
| Product page | thousands | medium | minutes | Cache it |
| Home page feed | hundreds | high | seconds | Cache it with a short TTL |
| Account balance | few | low | no | Do not cache, or cache with care |
| Search for a rare phrase | one | high | yes | A cache will mostly miss |
Common mistake
Adding a cache to fix a slow query that could be fixed with an index. A cache hides a slow query for the cached keys only. The first request for each key, and every request after a restart, still pays the full cost.
Check yourself:
- Why is a counter that changes on every click a poor candidate?
- A hot set is 3 GB and the cache is 1 GB. What problem should be expected?