Part 5 of 8 · 2 min read
Hot keys
One key so popular that a single cache server cannot carry it.
Think of a supermarket with ten checkouts. Customers spread across them. If a celebrity walks in and everyone wants to stand at checkout 3, the other nine are idle, and checkout 3 is swamped.
A cache server holds many keys, and each key lives on one server (or a small group). If a single key gets a very large share of traffic, the server that holds it is overloaded, even when the other servers are quiet.
Hot key
A hot key is a cache key that receives so many requests that the single server holding it becomes the bottleneck.
Example: a celebrity's profile during a live event is read 300,000 times a second. A cache server handles 100,000 requests a second.
A worked example. A cluster has 10 servers, each able to do 100,000 reads a second, so 1,000,000 in total. The hot key takes 300,000 a second, all landing on one server. That server needs three times its capacity. It slows, then fails, and its keys move to the neighbours (when hashing is consistent), which are then overloaded too.
Fixes:
- Local copy. Each application server keeps the hot key in its own memory for a few seconds (in-process cache, module 5). If there are 100 application servers and each reloads once a second, the cache server sees 100 reads a second for this key, not 300,000.
- Key splitting (replicas of the key). Store the value under several keys,
profile:42#1toprofile:42#8, on different servers. A reader picks one at random. With 8 copies, 300,000 ÷ 8 = 37,500 reads a second per copy. - Detect and promote. Count requests per key. When a key passes a threshold, switch to the local copy or key splitting.
| Fix | Reads/s hitting one cache server for the hot key (example) | Cost |
|---|---|---|
| None | 300,000 | Overload |
| Local copy, 100 app servers, 1 reload/s | 100 | Staleness of up to 1 second |
| 8 key copies | 37,500 | Writes must update 8 copies |
Check yourself:
- Why does adding more cache servers not fix a hot key?
- A hot key gets 240,000 reads a second. A server handles 80,000. How many key copies are needed at the least?