CSE Roadmap

Common tier · about 3 hours · 7 parts

Part 2 of 7 · 3 min read

The layers on the path

Every place a copy can be kept between the user and the data.

Think of a letter from a head office to a customer. A copy may be held by the customer, by the local post office, by the regional depot, by the head office clerk, and in the archive. A request for the same letter can be answered by the nearest holder.

The book "Building Microservices" splits caching into three groups. In client-side caching the client stores the result and decides when to fetch a fresh copy. With proxy caching, a proxy sits between client and server, such as a reverse proxy or a content delivery network. With server-side caching, the server handles it, perhaps with a system such as Redis or Memcached, or a simple in-memory cache. The author notes that for every public-facing website he has worked on, the result was a mix of all three.

The layers, from the user inwards:

Layer Where it runs What it holds Typical read cost
Browser cache The user's device Files and responses for that user 0 (no network)
CDN Servers near the user Static files and some pages, shared by many users Tens of ms
Reverse proxy In front of your servers Responses for many users A few ms
Application cache (in-process) Inside each server Objects, results of calculations Microseconds
Application cache (shared) A cache cluster Objects, sessions, query results About 0.5 ms
Database caches Inside the database and the operating system Pages of table and index data Memory speed

The database layer is easy to forget. The book "Designing Data-Intensive Applications" notes that even a disk-based storage engine may never read from disk if there is enough memory, since the operating system keeps recently used disk blocks in memory. Reading hot data from a table can already be a memory read before any cache of yours is added.

The CDN, from the book "System Design Interview", is a network of geographically spread servers used to deliver static content such as images, video, CSS and JavaScript. Level 1, module 4 covers it in full.

A request walks inwards until a layer can answer:

  1. The browser checks its own cache. A fresh copy is used with no network traffic at all.
  2. If not, the request goes to the CDN, which answers from a nearby server.
  3. If not, it goes to your reverse proxy or load balancer, and then to an application server.
  4. The application checks its in-process cache, then the shared cache.
  5. If both miss, it queries the database, which may answer from its own memory.
  6. The answer is stored on the way back in one or more of these layers.

Cache hierarchy (layers)

A cache hierarchy is a chain of caches on the path of a request. Each layer answers if it holds a fresh copy, and passes the request on if it does not.

Example: browser, CDN, application cache and database page cache, in that order.

Check yourself:

  • In which layer does a hit cost no network traffic at all?
  • Name two layers that are shared between many users and two that are not.