Part 4 of 7 · 2 min read
The numbers behind the gain
Why memory beats disk and the network by such a margin.
Think of fetching sugar from the shelf next to the stove versus the shop across town. The first takes seconds, the second takes a long walk. The latency table from Level 0 puts figures on the same gap.
| Operation | About |
|---|---|
| Read one value from memory | 100 nanoseconds |
| Round trip inside one data centre | 500 microseconds |
| Read 1 MB in order from a solid-state drive | 1 millisecond |
| Seek on a spinning disk | 10 milliseconds |
| Send a packet across an ocean and back | 150 milliseconds |
The book "System Design Interview" draws these conclusions from the table: memory is fast, the disk is slow, and disk seeks are to be avoided.
A cache wins in two ways at once:
- It keeps the data in memory, not on disk.
- If it sits close to the application, it saves a long trip. A cache in the same data centre costs a round trip of 500 microseconds, against 150 milliseconds for a distant server.
A worked example. A page needs 8 values. Each comes from a database on a spinning disk, at 10 ms per seek, so reading the values costs 8 × 10 = 80 ms in disk time alone. If all 8 are in a cache in the same data centre, each read costs one round trip of 0.5 ms, so 8 × 0.5 = 4 ms. The page is 20 times faster to assemble. If the cache is inside the application's own memory, each read takes about 100 nanoseconds, and 8 reads take under 1 microsecond.
Where a cache lives changes how much it saves:
| Place | Cost of one read (about) |
|---|---|
| Inside the application process | 100 nanoseconds |
| A cache server in the same data centre | 500 microseconds |
| The database (spinning disk) | 10 milliseconds or more |
| A server across an ocean | 150 milliseconds |
Computing is also worth caching. If a page needs a calculation that takes 200 ms of processor time, such as ranking 5,000 items, caching the result for a minute saves 200 ms for every other request in that minute. Caching saves the cost of recomputing, not only of reading.
Check yourself:
- A page does 5 database reads of 10 ms and 5 cache reads of 0.5 ms. How long do the reads take?
- Why can a cache inside the application be faster than a cache server, and what is the price?