Part 7 of 8 · 3 min read
Automatic or manual rebalancing
Who decides when to move data, and the danger of letting the system decide alone.
Automatic or manual rebalancing
Think of a thermostat that opens the windows when the room is too hot. If a sensor reads too high by mistake, the windows open, the heating works harder, and the sensor reads even hotter. A person watching might spot the fault.
Rebalancing can be fully automatic (the system decides when to move partitions without any administrator) or fully manual (an administrator configures the assignment, and it changes only when they say). Couchbase, Riak and Voldemort sit in between: they suggest an assignment and wait for an administrator to approve it.
The book weighs them. Automatic rebalancing means less operational work. But it can be unpredictable. Rebalancing is expensive: it reroutes requests and moves a large amount of data. Done carelessly, it can overload the network or nodes and harm other requests while it runs.
It is especially dangerous together with automatic failure detection. The book's story: one node is overloaded and slow to respond. The other nodes decide it is dead, and rebalance the cluster to move load away from it. That adds load on the overloaded node, the other nodes and the network, making things worse, and may cause a cascading failure. For this reason, a human in the loop can be good. It is slower, but it prevents surprises.
A worked example of the risk. A cluster of 10 nodes at 80 percent load. One node slows because of a bad disk. At 80 percent load on the other nine, moving its 10 percent of the data adds copying traffic of, say, 20 percent extra I/O (made up). The nine nodes cross 100 percent, slow down, and are declared dead in turn.
Limits that make automatic rebalancing safer:
- Throttle the data copy rate so that normal traffic is not starved.
- Move only a few partitions at once.
- Wait a long time after a node looks unhealthy before moving its data. A node that is rebooting returns soon and needs no rebalance.
- Require an administrator's approval for large moves.
Interview angle
Answer in your head first, then open it.
How do you search by a non-key field in a sharded database?
A local index (each shard indexes itself, query all shards, slow tail) or a global index (partitioned by the term, one lookup, but writes touch several shards and are often asynchronous).
How do you add a node without moving most of the data?
Not hash mod N. Use many fixed partitions, or dynamic splitting, or a ring. Only whole partitions move.
How many partitions should you create?
Enough for future nodes, but not so many that overhead hurts. Give the arithmetic: nodes × partitions per node.
Should rebalancing be automatic?
Not blindly. It is costly and can combine with failure detection to cause a cascade, so throttle it and keep an approval step.
Check yourself:
- What can go wrong when automatic rebalancing meets automatic failure detection?
- Name two ways to make rebalancing safer.
Words you will meet
| Word | Plain meaning |
|---|---|
| Secondary index | A lookup on a field other than the primary key |
| Local (document-partitioned) index | Each partition indexes only its own records |
| Global (term-partitioned) index | One index over all records, split by the term |
| Term | The value looked up in an index, such as colour:red |
| Scatter/gather | Sending a query to all partitions and combining the answers |
| Tail latency amplification | Many parallel calls make the slowest one decide the total time |
| Rebalancing | Moving data and load between nodes after the cluster changes |
| hash mod N | Placing a key on node hash mod number of nodes |
| Fixed partitions | A set number of partitions chosen at the start |
| Dynamic partitioning | Splitting and merging partitions as they grow or shrink |
| Pre-splitting | Creating an initial set of partitions on an empty database |