Part 3 of 8 · 2 min read
Skew and hot spots
What goes wrong when the split is unfair.
Think of 10 checkout lanes in a shop. If customers spread evenly, each lane has a short queue. If everyone queues at lane 1, nine lanes sit idle, and the shop is as slow as with one lane.
Skew and hot spot
Skew is an unfair split of data or load across partitions. A partition that gets far more load than the others is a hot spot.
Example: 10 partitions each hold 10 percent of the data, but one of them gets 60 percent of the requests. That partition is a hot spot.
The book explains: skew makes partitioning much less effective. In the extreme case all the load lands on one partition, nine out of ten nodes are idle, and the busy node is the bottleneck.
The simplest way to avoid hot spots is to assign records to nodes at random. The data spreads evenly, but when a record must be read, there is no way to know which node holds it, so all nodes must be queried in parallel. The better approach uses a rule from the key to the partition. The next two sections give two rules.
A worked example of the cost of skew. A cluster of 10 nodes can each do 5,000 writes per second, so 50,000 in total. The traffic is 40,000 writes per second.
- Even split: 4,000 per node. All nodes are below 5,000. Fine.
- 50 percent of the traffic goes to one partition: 20,000 writes per second to one node. It can do 5,000. The system fails at a total of only 10,000 writes per second (5,000 ÷ 0.5), a fifth of its nominal capacity of 50,000.
Skew has two sides:
- Data skew. One partition holds much more data. It may fill its disk first.
- Load skew. One partition gets much more traffic. It may run out of CPU first, even if the data is small.
Both need watching. The rules below deal with them in different ways.
Check yourself:
- Why does picking a node at random for each record make reads hard?
- 8 nodes can each do 2,000 writes per second. One partition takes 40 percent of the traffic. What total write rate can the system sustain?