CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 5 of 7 · 2 min read

Scaling

Which layer breaks first?

API tier. Stateless. Add servers behind the load balancer.

WebSocket tier. Stateful, since each server holds long connections. To remove a server, mark it as "draining" at the load balancer so no new connections arrive, wait for the existing ones to end, then remove it. A new software version needs the same care. A good load balancer does most of this.

User database. Profiles and friendships are too big for one server, so shard by user id (Level 4).

Location cache. 10 million active users × 100 bytes = 1 GB, which is small. But 334,000 writes a second is too much for one Redis server. The data per user is independent, so shard by user id across several servers, each with a standby node that can take over.

Pub/sub servers: the real bottleneck.

  • Memory: 100 million channels. Each channel keeps a small list of subscribers, about 20 bytes per subscriber. With 100 active friends per user: 100,000,000 × 100 × 20 bytes = 200 GB. With 100 GB servers, about 2 servers hold all channels.
  • CPU: 14 million pushes a second. A server may push about 100,000 messages a second (the book's cautious figure). So 14,000,000 / 100,000 = 140 servers.
  • The finding: CPU, not memory, decides. A cluster of about 140 servers is needed.

Spreading channels. Channels are independent, so share them across servers by hashing the user id, using a hash ring (consistent hashing, Level 4). Which server owns which channel is kept in a service discovery tool such as etcd or ZooKeeper. Each WebSocket server keeps a copy of the ring, and subscribes to changes, so it always knows where to publish and subscribe.

A worked example (made up). The ring has 4 pub/sub servers. User 17 hashes to server 2. Every handler that wants user 17's updates connects to server 2 for that channel. When server 2 dies, an operator (or automation) puts a fresh standby in its place on the ring. Service discovery notifies the WebSocket servers, and each handler re-subscribes only the channels that were on server 2.

Resizing is risky. The pub/sub servers look stateless, since messages are not stored. But each holds the subscriber lists, which makes them stateful. Adding or removing servers moves many channels, and every affected handler must re-subscribe at once. That causes a burst of work and some missed updates. So the cluster is kept over-sized, and resized only at the quietest time of day. Replacing one failed server is far cheaper, since only its channels move.

Friends added or removed. The client tells the WebSocket server, which subscribes to or unsubscribes from the friend's channel, and returns the friend's last location if the friend is active.

Read it as

Count messages, not storage. The first number to compute is the push rate, since it sets the size of the cluster.

Check yourself: 1. Which resource limits the pub/sub servers? 2. What does the hash ring decide? 3. Why is resizing the pub/sub cluster risky?