CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 2 of 7 · 2 min read

The cache inside each server

The simplest cache: a table in the memory of the application.

Think of every cook in a restaurant keeping a notepad of the most-used recipes at their own station. It is fast to read, and nobody else needs to be asked. But each cook's notepad is a little different, and a change to a recipe has to reach every pad.

The book "Understanding Distributed Systems" describes the simplest possible cache: an in-memory dictionary inside each client, such as a hash table with a size limit tied to the memory the node has.

In-process cache

An in-process cache lives in the memory of the application itself. Each application instance has its own, private copy.

Example: a web server keeps a map of the 5,000 most-read product pages in its own memory. A read costs about 100 nanoseconds.

Strengths:

  • It is the fastest cache: no network call, and no encoding of data.
  • It has nothing extra to run.
  • If one instance fails, other instances are unaffected.

Weaknesses, from the same book:

  • Inconsistency. Each cache is independent, so each client may see a different version of an entry.
  • Repeated fetches. An entry has to be fetched once per cache. The pressure on the database grows with the number of clients.
  • Cold starts and the thundering herd. When a service restarts or scales out, each new instance has an empty cache and goes straight to the database. A spike follows. The same happens when an item suddenly becomes popular.

A worked example of the second weakness. A product page is popular, and 50 application servers each cache it with a TTL of 60 seconds. Each server reloads it once a minute, so the database sees 50 reads a minute for this key. With one shared cache, the database sees about 1 a minute. For 100,000 popular keys, that is 5,000,000 reads a minute against 100,000.

A worked example of the third weakness. A service scales from 10 to 60 instances during a sale. Each new instance starts empty. If each needs to load 2,000 entries, the 50 new instances produce 100,000 database reads in their first minute.

The book offers request coalescing as a fix inside each process: at most one outstanding request per item per cache, so ten concurrent requests for one missing record send one request to the dependency.

Size limits matter. An in-process cache takes memory away from the application. A cache of 2 GB on a machine with 8 GB leaves 6 GB for the program. Memory pressure in the program itself (for example, garbage collection) can slow requests.

In-process cache
Speed Fastest (about 100 ns)
Consistency Weak: each instance has its own version
Load on the database Grows with the number of instances
Memory Competes with the application
Best for Small, hot, rarely changed data; a safety net in front of a shared cache

Check yourself:

  • 80 servers each cache a key with a TTL of 30 seconds. How many database reads per minute does this key cause?
  • Why do new instances create a spike on the database?