Part 4 of 8 · 2 min read
Components and storage
What runs where.
The book groups the system into three kinds of parts.
- Stateless services. Login, sign-up and profile. They sit behind a load balancer, as in earlier levels, and any server can answer any request. The key one is service discovery: it gives the client the address of a suitable chat server.
- Stateful service. The chat service. It is stateful because each client holds a persistent connection to one particular chat server, and stays there while it lives.
- Third-party integration. Push notification, as in module 6 of Level 10.
Other servers: presence servers track who is online, API servers do the ordinary work, and notification servers send push messages. A key-value store holds chat history.
Why a key-value store for messages? The book gives reasons. The data is enormous; the book mentions about 60 billion messages a day across two big apps. Mostly recent messages are read. Key-value stores scale out easily and give low delay. Relational databases handle the long tail of old data poorly, because large indexes make random reads costly. Real chat products use key-value stores: Messenger uses HBase and Discord uses Cassandra. Profiles and friend lists stay in a relational database, with replication and sharding.
Message tables. For one-to-one chat the key is the message id. For group chat the key is the pair (channel id, message id), and the channel id is the partition key, because every query is inside one group.
Message id. It must be unique and sortable by time. A creation timestamp is not enough, because two messages may arrive in the same moment. Options: a database counter, which key-value stores lack; a global generator such as snowflake (Level 10, module 2); or a local counter, unique inside one channel. A local counter is enough, because order only matters within one conversation, and it is simpler.
Common mistake
Ordering messages by a timestamp from the sender's phone. Clocks differ, and two messages can share the same time. Order by the id the server assigns.
Check yourself: 1. Why is the chat service stateful? 2. Why can a local counter be enough for message ids? 3. What is the partition key for group messages, and why?