Part 5 of 7 · 2 min read
Gossip and other options
Spreading changes between the nodes themselves, and finding the addresses.
Think of rumour in a village. Nobody keeps the official register, but each villager tells a few neighbours each day, and within a few days everyone knows the news. There is no central office to fail, though two villagers may hold different versions for a while.
The book describes a different approach, used by Cassandra and Riak: a gossip protocol among the nodes to spread any changes in cluster state. Requests can be sent to any node, and that node forwards them to the right node (approach 1). This puts more complexity in the database nodes, but avoids the dependency on an external coordination service like ZooKeeper.
Gossip protocol
In a gossip protocol, each node periodically tells a few randomly chosen other nodes what it knows about the cluster. The news spreads through the cluster like a rumour.
Example: node 4 learns that node 9 is down. It tells nodes 12 and 2. Each tells two more in the next round. After a few rounds all nodes know.
A worked example. 1,000 nodes. In each round, every node that knows tells 2 other random nodes. The number that know roughly triples each round (the knowing node plus 2 newly told): 1, 3, 9, 27, 81, 243, 729, then almost all. Seven or eight rounds suffice. With one round per second, the whole cluster learns in about 8 to 10 seconds. (Real counts are lower because some pushes hit nodes that already know, but the growth is exponential, so the time grows with the logarithm of the cluster size.)
Other points from the book:
- Couchbase does not rebalance automatically, which simplifies the design. It is normally set up with a routing tier called moxi, which learns about routing changes from the cluster nodes.
- When using a routing tier or sending to a random node, clients still need the IP addresses to connect to. These change less often than the partition assignment, so DNS is often enough.
| Mechanism | Strength | Weakness |
|---|---|---|
| Coordination service | One consistent source of truth, instant notification | Extra system to run; must be highly available |
| Gossip | No external dependency, no single point of failure | Map is only eventually consistent; extra complexity in nodes |
| Static configuration | Simple | Cannot cope with change |
| DNS | Simple, works everywhere | Slow to change (caching); only for addresses |
Stale maps in practice. Whatever the mechanism, some requests will use an old map. A good design handles it:
- The old owner answers "I no longer own this partition" or forwards the request.
- The caller refreshes its map and retries.
- During a partition move, the old assignment serves requests until the transfer is complete (module 5).
Common mistake
Hard-coding the list of nodes in clients. After the first rebalance or failure, clients keep calling the wrong nodes. Use a map that is updated, and a retry on "not the owner".
Check yourself:
- How does gossip reach all nodes without a central store?
- Why is DNS enough for finding node addresses, but not for partition ownership?