CSE Roadmap

Specialist tier · about 4 hours · 8 parts

Part 4 of 8 · 3 min read

Flows

Sending, receiving and the parts.

Parts.

  • Web servers: public request and response servers for sign-up, profiles and all email calls (send, list a folder, read a message).
  • Real-time servers: stateful servers that push new mail to online clients over WebSocket (Level 11, page 2). If a browser cannot use WebSocket, fall back to long polling.
  • Metadata database: subjects, bodies, senders, recipients and flags.
  • Attachment store: object storage (Level 12, page 3). Large files do not belong in the metadata database.
  • Distributed cache: keeps the most recent mails in memory, since users read those again and again.
  • Search store: full-text search (section 5).
  • Queues: an outgoing queue and an incoming queue.
  • SMTP servers and workers: outgoing workers send mail, and incoming SMTP servers accept it.

Sending flow.

  1. The user presses send. The request goes to the load balancer.
  2. The balancer applies rate limits and routes to a web server.
  3. The web server does basic checks, such as the size limit. If the recipient is in the same domain, it also checks for spam and viruses and stores the mail straight into the sender's Sent folder and the recipient's Inbox.
  4. Otherwise it puts the message on the outgoing queue. A very large attachment goes to the object store, and the queue message keeps a reference. A failed check goes to an error queue.
  5. SMTP outgoing workers read the queue, check for spam and viruses again, and store the message in the Sent folder.
  6. The workers send the message to the recipient's mail server.

The queue decouples web servers from SMTP workers, so each scales alone. The queue length is watched. A long queue means the recipient server is down (retry later) or there are too few workers (add more).

Receiving flow.

  1. Another server's SMTP connection reaches the load balancer, which spreads it over incoming SMTP servers. Acceptance rules apply at connection time, so invalid mail is turned away early.
  2. A large attachment goes to the object store first.
  3. The message goes onto the incoming queue, which buffers surges.
  4. Mail processing workers filter spam, stop viruses and check the message.
  5. The message is stored in the mail store, the cache and the object store.
  6. If the recipient is online, the message is pushed through the real-time servers.
  7. If offline, it waits in storage. When the client returns, the web servers fetch the new mail.

A worked example (made up). At peak, 300,000 messages a second arrive for 4 seconds, while workers handle 150,000 a second. The queue grows by 150,000 × 4 = 600,000 messages. When arrivals drop to 75,000 a second, the queue shrinks by 75,000 a second and empties in 8 seconds. No mail is lost, and nobody waits more than about 12 seconds.

Dead letter and error queue

An error queue holds messages that failed a check, so that nothing is lost and a person can look at them.

Example: a message over the size limit goes to the error queue, and the sender gets a bounce.

Check yourself: 1. Why a queue between web servers and SMTP workers? 2. Where do large attachments go? 3. What happens to mail for an offline user?