CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 3 of 7 · 2 min read

Hit, miss and the hit ratio

The measure of how well a cache works, and the average time that follows from it.

Think of the counter and the storeroom again. If 9 out of 10 trips are satisfied from the counter, the cook is fast on average. If only 5 of 10 are, the cook still walks to the storeroom half the time.

A cache hit happens when the requested item is found in the cache. A cache miss happens when it is not, and the data must be fetched from the slower source. The book "Understanding Distributed Systems" uses the same two words.

Hit ratio

The hit ratio is the number of hits divided by the number of requests, as a fraction or a percentage.

Example: 900 hits out of 1,000 requests is a hit ratio of 0.9, or 90 percent. The miss ratio is the rest, 10 percent.

The average read time follows from the hit ratio. A hit costs the cache time. A miss costs the cache lookup and then the slow source (the lookup is wasted), and then the data is stored.

Average time = hit ratio × cache time + miss ratio × (cache time + source time)

A worked example, with made-up times: the cache answers in 1 ms and the database in 50 ms.

Hit ratio Calculation Average time
0 percent 0 × 1 + 1.00 × 51 51 ms
50 percent 0.5 × 1 + 0.5 × 51 26 ms
90 percent 0.9 × 1 + 0.1 × 51 6 ms
99 percent 0.99 × 1 + 0.01 × 51 1.5 ms

Going from 90 to 99 percent cuts the average time from 6 ms to 1.5 ms, a factor of four. The last few points of hit ratio matter a great deal, since each miss is expensive.

The second benefit is load removed from the source. A database that can serve 2,000 reads per second is overwhelmed by 10,000 reads per second. A cache changes how many reads reach it:

Hit ratio Reads per second reaching the database (of 10,000)
80 percent 2,000
90 percent 1,000
99 percent 100

At 80 percent the database is at its limit. At 99 percent it has 20 times more room. The book "System Design Interview" names this as a reason for a cache tier: better performance, less load on the database, and the ability to scale the cache on its own.

A hit ratio is a number to measure in production, not to guess. A dashboard that shows hits, misses and the hit ratio per cache is the first tool for tuning it. A falling hit ratio is an early warning that a key space has changed or that the cache is too small.

Check yourself:

  • A cache serves 7,200 hits and 800 misses in a minute. What is the hit ratio?
  • With a 1 ms cache and a 20 ms source, what is the average read time at a 95 percent hit ratio?