Part 2 of 7 · 2 min read
The cache inside each server
The simplest cache: a table in the memory of the application.
Think of every cook in a restaurant keeping a notepad of the most-used recipes at their own station. It is fast to read, and nobody else needs to be asked. But each cook's notepad is a little different, and a change to a recipe has to reach every pad.
The book "Understanding Distributed Systems" describes the simplest possible cache: an in-memory dictionary inside each client, such as a hash table with a size limit tied to the memory the node has.
In-process cache
An in-process cache lives in the memory of the application itself. Each application instance has its own, private copy.
Example: a web server keeps a map of the 5,000 most-read product pages in its own memory. A read costs about 100 nanoseconds.
Strengths:
- It is the fastest cache: no network call, and no encoding of data.
- It has nothing extra to run.
- If one instance fails, other instances are unaffected.
Weaknesses, from the same book:
- Inconsistency. Each cache is independent, so each client may see a different version of an entry.
- Repeated fetches. An entry has to be fetched once per cache. The pressure on the database grows with the number of clients.
- Cold starts and the thundering herd. When a service restarts or scales out, each new instance has an empty cache and goes straight to the database. A spike follows. The same happens when an item suddenly becomes popular.
A worked example of the second weakness. A product page is popular, and 50 application servers each cache it with a TTL of 60 seconds. Each server reloads it once a minute, so the database sees 50 reads a minute for this key. With one shared cache, the database sees about 1 a minute. For 100,000 popular keys, that is 5,000,000 reads a minute against 100,000.
A worked example of the third weakness. A service scales from 10 to 60 instances during a sale. Each new instance starts empty. If each needs to load 2,000 entries, the 50 new instances produce 100,000 database reads in their first minute.
The book offers request coalescing as a fix inside each process: at most one outstanding request per item per cache, so ten concurrent requests for one missing record send one request to the dependency.
Size limits matter. An in-process cache takes memory away from the application. A cache of 2 GB on a machine with 8 GB leaves 6 GB for the program. Memory pressure in the program itself (for example, garbage collection) can slow requests.
| In-process cache | |
|---|---|
| Speed | Fastest (about 100 ns) |
| Consistency | Weak: each instance has its own version |
| Load on the database | Grows with the number of instances |
| Memory | Competes with the application |
| Best for | Small, hot, rarely changed data; a safety net in front of a shared cache |
Check yourself:
- 80 servers each cache a key with a TTL of 30 seconds. How many database reads per minute does this key cause?
- Why do new instances create a spike on the database?