CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 7 of 8 · 3 min read

State that stays, and how to route to it

Connections and rooms that must stay on one machine, and the routing rule that keeps the rest flexible.

State that stays, and how to route to it

Think of a phone call. The two phones are joined for the whole call. If the line is cut, the call ends, and no other line can continue it.

Some work is stateful by nature.

  • Long-lived connections. A chat server holds an open connection for each user. Messages are pushed down that connection.
  • In-memory game rooms or collaborative documents. Many users edit one thing, and one process keeps the live copy.
  • Streaming and large uploads in progress. A half-sent file is held until the rest arrives.

These parts should be few and clearly marked. The rest of the system stays stateless around them. The usual approach:

  1. Keep the durable truth in a store. The chat server holds connections, but messages are saved in a database. If a chat server dies, users reconnect to another one and read what they missed.
  2. Route by a key. The balancer or a lookup service maps a key (the room id, the document id) to a server. A hash does this simply. The book describes consistent hashing for this job, so that adding or removing a server moves few keys. Level 4 explains it in full.
  3. Make loss survivable. On a crash, clients reconnect, and the room is rebuilt from the store.

A worked example. A chat system has 10 servers and 5 million connected users, so 500,000 connections per server on average (5,000,000 ÷ 10).

  • If one server fails, 500,000 clients reconnect at the same time. That is a burst on the other 9, which each take about 55,556 new connections (500,000 ÷ 9).
  • Clients should wait a random time before they reconnect, so that all do not return in the same second. A random delay up to 30 seconds spreads 500,000 connections to about 16,700 per second on average (500,000 ÷ 30).
  • A server that restarts holds nothing that the store lacks, so recovery is quick.

Single point of failure

A single point of failure is a part whose failure stops the whole system.

Example: one session store with no copy. If it fails, nobody can log in.

Each stateful part is a candidate. A design that moves state out of servers has not removed the state. It has concentrated it in a few places that are built to survive, with copies and backups.

Interview angle

Answer in your head first, then open it.

Why make the web tier stateless?

Any server can answer any request, so a balancer can spread load freely, servers can be added or lost without harm, and autoscaling works.

Where do you keep the session?

In a shared store with an expiry, or in a signed token that has a short life. Say the trade-off: lookups against revocation.

Do you use sticky sessions?

Only as a speed-up that can be lost without damage. Correctness must not depend on them.

What stays stateful in your design?

Name the connections or rooms, say that the truth is saved in a store, and say how the routing finds the right server.

Check yourself:

  • In a chat system, what is saved in the database, and what lives only in the server?
  • Why should clients wait a random time before reconnecting after a server crash?

Words you will meet

Word Plain meaning
State Information kept from one request to the next
Stateless A server that keeps nothing between requests
Cookie A small text a browser stores and sends back
Session The saved facts about one logged-in visit
Session store A shared place where sessions are kept
Signed token A message with facts and a proof that the server made it
Signature A value computed with a secret key that shows no one changed the message
Refresh token A long-lived token that is traded for a new short-lived one
Revocation Cancelling something already issued
Sticky session A rule that sends a user to the same server each time
Autoscaling Adding and removing servers automatically with load
Single point of failure A part whose failure stops everything