CSE Roadmap

Specialist tier · about 4 hours · 8 parts

Part 6 of 8 · 2 min read

Deliverability and search

Getting mail into the inbox, and finding it.

Deliverability. Sending a message is easy. Landing in the inbox is hard. More than half of all mail is spam, so big providers distrust new senders.

  • Dedicated IP addresses for sending.
  • Separate IPs by category, so marketing mail does not harm the reputation of important mail.
  • Warm up new addresses slowly. The book quotes 2 to 6 weeks.
  • Ban spammers fast to protect the reputation.
  • Process feedback. A hard bounce means the address is invalid. A soft bounce is a temporary failure such as a full mailbox or a busy server, and the message is retried. A complaint means the recipient pressed "report spam". Each goes to its own queue so it can be handled separately.

Authentication. Phishing is behind most breaches. Three standards help receivers trust the sender: SPF (which servers may send for a domain), DKIM (a signature on the message) and DMARC (a policy for what to do when the checks fail).

Search. Mail search differs from web search. It has many writes (every send, receive or delete changes the index) and few reads, since a query runs only when a user presses search. Each user only searches their own mail.

Two options:

Option Gain Cost
Search engine (such as Elasticsearch), with user id as the routing key so one user's data sits together Easy to add, good full-text search Must keep in sync with the primary store, more machines, may need a dedicated team at very large scale
Native search built into the mail database, with index data organised as a log-structured merge-tree Sequential writes and a tuned design, fewer syncing problems Much engineering work

An LSM tree (log-structured merge-tree) takes new entries into memory first and merges them into larger sorted levels on disk later. It writes sequentially, which suits write-heavy indexes. Data that changes often (folder labels) can be kept apart from data that never changes (the mail itself).

A reasonable rule: a small or medium service uses a search engine and rebuilds its index from the primary store if it is lost. A very large provider builds search into its own database.

Reindexing runs outside the main path. When mail is sent, received or deleted, an event goes on a queue (Level 12, page 1) and a job updates the index. Search itself is a synchronous call, since the user waits for it.

Hard bounce

A hard bounce is a permanent delivery failure, such as an address that does not exist.

Example: mail to "[email protected]" bounces for good, so the sender stops using that address.

Check yourself: 1. Why must a new sending IP be warmed up? 2. Name the three feedback types. 3. How do write patterns of email search differ from web search?