Part 3 of 7 · 2 min read
The shared cache server
A separate cache service that all application servers use.
Think of a single noticeboard in the corridor that every cook reads. One change to the board is seen by all. The board is a bit further away than a notepad on the counter, and if the board falls, everybody notices.
The book "Understanding Distributed Systems" describes the external, out-of-process cache: shared by all service instances, which solves some of the drawbacks of the in-process cache at the price of more complexity and cost.
Out-of-process (shared) cache
An out-of-process cache runs as a separate service. All application instances read and write it over the network.
Example: ten web servers use one Memcached cluster. A read for product:91 takes a round trip of about 500 microseconds inside the data centre.
Strengths, according to the book:
- There can be only one version of each item at a time. It may be out of date, but every client sees the same version, so inconsistency drops.
- The load on the dependency no longer grows with the number of clients.
Weaknesses:
- It is another service to maintain and operate.
- The latency is higher than that of an in-process cache, since a network call is needed.
- The load has only moved. If it grows, the cache itself must scale out.
"System Design Interview" shows a web server using cache servers through simple calls: set a key with a value and an expiry, get a key. Cache servers provide such calls for common programming languages.
Memcached and Redis. These are two well-known cache servers.
- The book "Designing Data-Intensive Applications" says that some in-memory key-value stores, such as Memcached, are meant for caching only, where it is acceptable for data to be lost when a machine restarts. Others aim for durability by writing a log to disk, taking snapshots, or replicating state to other machines. It says Redis and Couchbase provide weak durability by writing to disk asynchronously, and that Redis also offers data structures, such as priority queues, as a database-like interface.
- For a cache, treat both as memory stores whose contents may vanish. Anything that must survive goes in the database.
Compare the two places by a worked example. A page makes 6 cache reads.
| Place | Cost per read | Total for 6 reads |
|---|---|---|
| In-process | 100 ns | 0.0006 ms |
| Shared server | 0.5 ms | 3 ms |
| Database | 10 ms | 60 ms |
Both caches beat the database by a wide margin. The in-process cache is 5,000 times faster than the shared one, but that gap matters only for pages with many reads.
Use both together: a small in-process cache with a short TTL (1 to 5 seconds) in front of a shared cache, which sits in front of the database. The in-process layer takes the hot keys (module 4) and gives some protection if the shared cache is down.
Check yourself:
- Why does every client see the same version with a shared cache, even when it is out of date?
- What does a shared cache cost that an in-process cache does not?