CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 4 of 7 · 2 min read

The numbers behind the gain

Why memory beats disk and the network by such a margin.

Think of fetching sugar from the shelf next to the stove versus the shop across town. The first takes seconds, the second takes a long walk. The latency table from Level 0 puts figures on the same gap.

Operation About
Read one value from memory 100 nanoseconds
Round trip inside one data centre 500 microseconds
Read 1 MB in order from a solid-state drive 1 millisecond
Seek on a spinning disk 10 milliseconds
Send a packet across an ocean and back 150 milliseconds

The book "System Design Interview" draws these conclusions from the table: memory is fast, the disk is slow, and disk seeks are to be avoided.

A cache wins in two ways at once:

  1. It keeps the data in memory, not on disk.
  2. If it sits close to the application, it saves a long trip. A cache in the same data centre costs a round trip of 500 microseconds, against 150 milliseconds for a distant server.

A worked example. A page needs 8 values. Each comes from a database on a spinning disk, at 10 ms per seek, so reading the values costs 8 × 10 = 80 ms in disk time alone. If all 8 are in a cache in the same data centre, each read costs one round trip of 0.5 ms, so 8 × 0.5 = 4 ms. The page is 20 times faster to assemble. If the cache is inside the application's own memory, each read takes about 100 nanoseconds, and 8 reads take under 1 microsecond.

Where a cache lives changes how much it saves:

Place Cost of one read (about)
Inside the application process 100 nanoseconds
A cache server in the same data centre 500 microseconds
The database (spinning disk) 10 milliseconds or more
A server across an ocean 150 milliseconds

Computing is also worth caching. If a page needs a calculation that takes 200 ms of processor time, such as ranking 5,000 items, caching the result for a minute saves 200 ms for every other request in that minute. Caching saves the cost of recomputing, not only of reading.

Check yourself:

  • A page does 5 database reads of 10 ms and 5 cache reads of 0.5 ms. How long do the reads take?
  • Why can a cache inside the application be faster than a cache server, and what is the price?