CSE Roadmap

Specialist tier · about 5 hours · 8 parts

Part 6 of 8 · 3 min read

Architecture

One owner per document.

Parts.

  • Clients: web or mobile editors, with a local copy, a queue of pending operations and the transformation logic (OT) or merge logic (CRDT).
  • Gateway: terminates WebSockets (page 1), authenticates, and routes each connection to the server that owns the document.
  • Document session servers: each handles many documents. For each document it keeps the current state and recent operations in memory, orders and transforms operations, and broadcasts. All editors of one document connect (through the gateway) to the same server. That is how ordering stays simple.
  • Routing: a mapping from document id to the owning server, kept in a coordination service (page 6) or computed by consistent hashing (Level 3) over the servers. If a server dies, ownership of its documents moves to others.
  • Operation log: every operation is appended to a durable log (a replicated log or a database table keyed by document and revision) before it is acknowledged.
  • Snapshots: periodically (say every 500 operations or 10 minutes) save the whole document. Opening a document loads the last snapshot and replays the operations after it. Snapshots also bound the log.
  • Document store: snapshots and logs in a database partitioned by document id (Level 3).
  • Presence service: cursors, selections and "who is here". It changes fast and need not be durable. Keep it in memory, send it through the same WebSocket, and drop old entries when a user disconnects (a heartbeat).
  • Comments and permissions: ordinary services with a database. Comments anchor to a place in the text, which must itself be adjusted by operations (an anchor moves like a cursor).
  • Version history: named versions are snapshots, and any earlier state is a snapshot plus a replay up to that revision.

Flow of one edit.

  1. The user types. The client applies it locally at once and sends it with its base revision.
  2. The gateway forwards it to the owning server.
  3. The server transforms it, assigns a new revision and appends it to the log.
  4. After the log write succeeds, it acknowledges the sender and broadcasts to the others.
  5. Other clients apply it and update the screen.

Failure and recovery. If the owning server dies, a new owner loads the last snapshot and the log, rebuilds the state and carries on. Clients reconnect and send the revision they have. The server sends them the operations they missed. Clients resend pending operations, which the server accepts only once (using an operation id, so a repeat is ignored).

Offline editing. The client keeps working locally and stores pending operations. On reconnection it sends them with the old revision. OT must transform them against everything since (possibly a long list), and a CRDT merges them. Long offline periods favour CRDTs.

Scaling.

  • Partition documents over servers by id. A very popular document (a live event with 10,000 viewers) separates editors (few, through the owning server) from viewers (many, served through read-only relays that receive the broadcast from the owner).
  • Rate limit operations per user, and cap the size of an operation, to protect the server.

A worked example (made up). A server handles 20,000 documents with 1.5 operations a second each: 30,000 operations a second. At 100 bytes each, that is 3 MB/s of log writes. 1 million operations a second over the whole fleet needs about 34 such servers (1,000,000 / 30,000), so 40 with spare capacity.

Read it as

One owner per document gives a single order of edits. Everything else (logs, snapshots, presence) supports that owner.

Check yourself: 1. Why must all editors of one document reach the same server? 2. What does a snapshot save? 3. How are duplicate operations from a reconnecting client avoided?