How to design an email service

An email service has to send, receive, store, and search mail for a billion mailboxes. The whole design turns on one fact: a message is submitted once but delivered to several recipients, so its body is stored a single time while its search index is built once per recipient.

In this lesson, we’ll build that service outward from that one fact, sizing each tier from the workload before we commit to a mechanism. By the end you’ll be able to:

• size the stored mail, and the search index over it, from first principles;
• explain why the same body is stored once but the index is built once per recipient;
• state what the SMTP protocol does and does not promise about delivery;
• design outbound sending against the reputation failures that take mail systems down.

What goes in and what comes out

Before we size anything, let’s be precise about what the service takes and what it returns. There are two inputs and two outputs:

• Inbound. A raw message arrives from another mail server. Out of it come three things: one stored copy of its bytes, one small row per recipient mailbox, and one batch of search-index entries per recipient. Only after all three are safely on disk does an acceptance code go back to the sender.

• Outbound. A user submits a message through the API. Out of it comes a queued item per destination domain, retried for days. The eventual outcome arrives later as a separate inbound message, not as the response to the submission.

So inbound is one stored lump of bytes (a blob) with many small rows referring to it, and outbound gives no immediate answer at all.

Why this is a storage problem

Email looks like a messaging problem, but it is really a storage and retrieval problem behind a delivery protocol. Two numbers decide the architecture before anything else:

• The corpus (the complete body of stored mail) reaches exabytes if you keep one copy per recipient.
• The search index over that corpus is the second-largest thing you own.

Two size units recur below. A petabyte (PB) is a million gigabytes. An exabyte (EB) is a thousand petabytes.

Three common mistakes

• Storing a copy per recipient, which multiplies the corpus by the fanout: the average number of mailboxes one submitted message is delivered to.
• Reaching for “we add Elasticsearch” (a popular open-source search engine) without sizing an index or asking whose messages are in it.
• Treating delivery status as something an API call returns. It is not. The protocol is store-and-forward: each server accepts the message, takes responsibility for it, and passes it on later, retrying for days. Failures come back out of band, as new mail, not as a response.

Ideas borrowed from other chapters

Each is restated in one line where it is used, so nothing here depends on reading them first.

• The estimation chapter: the rounding trick of treating a day as 100,000 seconds instead of 86,400, which turns a per-day number into a per-second one by shifting the decimal.
• The file storage chapter: blob storage, content addressing (naming stored bytes after a hash of themselves), and the privacy side channel that sharing one copy between users opens.
• The message queue chapter: outbound queueing and retry behaviour.
• The database internals chapter: how much extra disk writing log-structured storage does, and how to choose the column that decides which server a row lives on.

Framing: what decision, and what breaks

The decision: where the fanout lands

A message submitted once is delivered to many mailboxes, and every subsystem either pays that multiplier or dodges it. Storage dodges it: one blob (a single opaque lump of bytes stored under a name) with many references pointing at it. The search index does not dodge it, and that is a deliberate choice priced out later.

The four properties, and what each costs

Each guarantee the service makes forces machinery to exist.

Four terms from that table run through the chapter:

• 250 is the SMTP success code: the receiving server saying “I have this message and I take responsibility for it”. Durability has to be reached before it is sent.
• Content addressing means naming a stored object by a hash of its own bytes. Identical content then shares one name and one copy with no coordination.
• Reference counting and a garbage collector are how you learn that the last mailbox pointing at a blob has gone away and the bytes may be reclaimed.
• Deliverability is the fraction of your mail that reaches an inbox instead of a spam folder. Other companies decide it by scoring your sending behaviour, which is why reputation is a subsystem, not a footnote.

The core design fits in one sentence: the same body is stored once and referenced by every recipient, but the search index is per-user, so the fanout is paid exactly once and you choose where.

Three things that break in production

• A mailing-list blast. One submission turns into a million mailbox rows and a million index updates.
• A compromised account. It sends spam from a shared outbound IP address, and every other customer’s mail on that address starts landing in junk folders.
• A large receiving provider goes down. The outbound queue grows for four days, because four days of retrying is what the protocol asks for.

Each of these three is a measurement crossing a line, so before we design anything we pin down what the service must guarantee and then size the load that tests those guarantees.

Requirements

SMTP is the Simple Mail Transfer Protocol, the language mail servers speak to each other; a server that speaks it is a mail transfer agent (MTA).

Functional

• Receive mail over SMTP from the internet; accept, defer, or reject.
• Send mail: authenticated submission, queueing, retry, and bounce handling.
• Read: list a mailbox by label, read a message, download attachments.
• Search a user’s own mail by keyword, sender, date, and label.
• Label, star, mark read, archive, delete; sync those across devices.
• Classify spam and enforce sending reputation.

Three things are out of scope:

• Calendaring and contacts: separate products that share a login.
• End-to-end encryption, where only the two humans hold the keys. Excluded from the main design because it destroys deduplication (priced in the alternatives section).
• The mail client that renders the message, its own product.

Non-functional

Each target forces a specific mechanism later.

Some shorthand from that table:

• p99 is the ninety-ninth percentile: the figure 99 out of 100 requests come in under. It describes the slow tail an average would hide.
• Eventually consistent means parts of the system may disagree for a while but converge once changes stop.
• Immutability means the stored bytes are never edited in place. Anything a user changes lives somewhere else.

The consistency contract, stated precisely: a user must never see a message vanish, and must never see one they were told arrived fail to be searchable within seconds. Everything else (counts, label totals, thread ordering during a rebuild) may lag.

Back of the envelope

A back-of-the-envelope estimate is a deliberately rough calculation in round numbers. Three follow: how much mail flows, how much you actually store, and what that costs in physical disk.

The shared assumptions:

1 B mailboxes, 300 M daily actives, 30 received and 5 sent per active per day,
10 KB of headers and body, 20% carry a 300 KB attachment, peak = 3x average

Daily actives are the users who open the product on a given day, far fewer than accounts, because most accounts are dormant. QPS is queries (operations) per second.

Flow

Delivered mail is 300M x 30 = 9 B/day, which at 100,000 seconds/day is 90,000 delivery QPS (270,000 at peak). Submissions are 300M x 5 = 1.5 B/day, or 15,000 QPS.

The central number is the average in-system fanout: 9 B delivered / 1.5 B submitted = 6. Every submitted message becomes six delivered messages you must store. In-system matters: it counts only mailboxes you host, the only copies you would store.

The bytes per message is a weighted average, not a typical message: 10,000 + 0.20 x 300,000 = 70,000 bytes. No message is actually 70 KB; most are 10 KB and one in five is 310 KB.

Storage, and what dedup is worth

The fanout of 6 is exactly the multiplier we want to refuse to pay on bytes. Let’s price out what refusing it is worth. Deduplication means storing one copy of identical content and pointing every user at it. It applies at two levels (whole messages and individual attachment parts) and the two savings multiply.

• One copy per recipient: 9 B x 70 KB = 630 TB/day, which over five years is 1.15 EB.
• One copy per submission: 1.5 B x 70 KB = 105 TB/day. Message-level dedup therefore saves 525 TB/day, 83.3% of everything written. That fraction is exactly the fanout by construction: dividing by 6 removes 5/6, and 5/6 = 0.833.
• Attachment dedup: of the ~90 TB/day of attachment bytes in the unique stream, ~30% are content-hash duplicates, removing about 27 TB/day.
• Net stored: 78 TB/day, or 142 PB over five years against the naive 1.15 EB.

The extreme case makes it intuitive: a single 70 KB message to 1,000 recipients is 70 MB stored per-recipient versus 70 KB stored once, 99.9% redundant. The mean fanout of 6 gives 83%; the tail is where per-recipient storage becomes indefensible.

What dedup does not collapse

Sharing the bytes does nothing for per-recipient state: whether a user has read the message, which labels they filed it under, whether they deleted it, where it sits in their thread. That state is the metadata tier: the small mutable rows describing one user’s relationship to a message.

One mailbox row is about 124 bytes (user id, message id, thread id, timestamp, flags, labels, blob reference, overhead). At 9 B rows/day that is 1.12 TB/day, ~2 PB over five years, just 1.06% of the content written, but 100% of what a user can change.

That split is what makes the two-tier design work. The blob tier is immutable, eventually consistent, and cheap. The metadata tier is mutable, strongly consistent, and sharded on user_id: split across many database servers, with the user’s identity deciding which server holds their rows, so every query about one user touches exactly one machine.

Physical layout

Logical bytes are not disk bytes; you store redundantly. Two mechanisms:

• A replica is one complete copy on a different machine. Three replicas survive disk and machine failures at 3x the size.
• Erasure coding stores the data plus recovery information spread across machines, surviving comparable failures at roughly 1.4x the size. The price is slower reads, since reconstructing a lost piece means fetching from several machines. So erasure coding suits the cold tail and replication suits recent mail.

Split the corpus into a hot tier (last 30 days, 3 replicas) and a cold tier (everything older, erasure coded at 1.4x):

• Hot: 78 TB/day x 30 x 3 ≈ 7 PB.
• Cold: ~140 PB x 1.4 ≈ 196 PB.
• Physical total ≈ 203 PB, against a naive 1.15 EB x 3 = 3.45 EB.

That is a 17x reduction, and it factors cleanly: 8.08x from dedup (1.15 EB / 142 PB) times 2.10x from the storage policy (427 PB replicated-throughout / 203 PB tiered), and 8.08 x 2.10 ≈ 17. Neither factor alone gets there.

A sanity check on one mailbox

Dividing back down to one user catches factor-of-1000 errors. One active mailbox receives 30 messages a day for 5 years, = 54,750 messages at 70 KB each ≈ 3.83 GB, against a typical 15 GB quota. The quota binds almost nobody, which matches reality and confirms that enforcing quotas is a billing feature, not a capacity mechanism.

Which assumptions matter

Every number rests on an assumption. These are the load-bearing ones: wrong by an order of magnitude and a conclusion falls over.

Three of these are worth confirming with whoever owns the product, because they change a decision, not just a digit: retention, the spam share, and the outbound IP count. If the real fanout were 2 instead of 6, the per-user index would cost a third as much and the argument would get easier.

API sketch

The two interfaces serve different callers. The SMTP surface is machine-to-machine: other organisations’ mail servers speak it to hand you mail. The HTTP surface is what the user’s own mail client calls.

MAIL FROM, RCPT TO, and DATA are the three commands of an SMTP conversation, carrying the sender, each recipient, and the message bytes, in that order.

SMTP  25   inbound   MAIL FROM / RCPT TO / DATA -> 250 only after the blob and
                     every recipient's mailbox row are durable
SMTP  587  submission  authenticated, per-account rate limited, DKIM signed here

GET  /v1/mailbox/{label}?cursor=&limit=50 -> [{message_id, thread_id, snippet,
                                               from, received_at, flags}], cursor
GET  /v1/messages/{message_id}            -> headers, body, part manifest
GET  /v1/parts/{content_hash}             -> 302 to a signed, expiring blob URL
POST /v1/messages:send                    Idempotency-Key: ...
       {"to": [...], "subject": "...", "body": "...", "parts": [content_hash]}
POST /v1/messages/{message_id}:modify     {"addLabels": [...], "removeLabels": [...]}
GET  /v1/search?q=&cursor=                -> this user's index only, never global
GET  /v1/history?since=                   -> the per-user change journal

Port 25 accepts mail from the rest of the internet; anyone may connect, so it assumes every caller is hostile. Port 587 is the submission port, where authenticated users hand you mail to send. Only this port signs outgoing mail with DKIM: a cryptographic signature over the message that lets any receiver verify it really came from your domain and was not altered in transit.

Four choices in that sketch carry weight:

• 250 is returned after durability, never before. The moment you answer 250 the sending server may delete its copy. That response is the only durability contract in the protocol, and it cannot be taken back.
• Attachments are fetched by content hash, not by message id. The same 2 MB slide deck forwarded to you is the same bytes. Naming it by a hash of its content makes the cache, the content delivery network (geographically distributed copies serving nearby users), and the dedup index agree with no coordination.
• Search takes no user parameter. The caller’s identity selects which index is read. A search API that accepts a mailbox id is one authorization bug away from leaking one account’s mail to another. Partitioning turns that from a check the code must remember into a property of the data.
• :modify sends a list of labels, not a rewritten message. Bodies never change; everything a user can change lives in the small mailbox row.

Sending carries an idempotency key: a client-chosen string that lets the server recognise a retry of the same intent. Without it, a retried submission sends the mail twice, which the recipient sees and nothing can undo (the same mechanism a hotel booking uses, in the hotel reservation chapter).

Data model

Three decisions inside the schema hold up the design: what is sharded and what is not, why blob references are not counted transactionally, and why a user’s search index is stored as ordinary rows.

This is a column list per table, not runnable SQL: one row per user per message, one record per distinct message, one record per distinct attachment, then four supporting tables.

mailbox                                   -- sharded on user_id
  user_id BIGINT, msg_seq BIGINT,         -- PK (user_id, msg_seq), monotonic
  message_id BYTEA(16), thread_id BIGINT,
  blob_ref BYTEA(32),                     -- content hash of the immutable message
  received_at TIMESTAMP, flags SMALLINT, labels BIGINT

messages                                  -- global, content-addressed, immutable
  message_id PK, envelope, header_blob, body_blob,
  part_manifest,                          -- [(content_hash, size, mime, filename)]
  recipient_count INT, first_seen

parts     content_hash BYTEA(32) PK, size BIGINT, mime_type, blob_locator, first_seen
segments  user_id, segment_id, min_seq, max_seq, dict_offset, postings_locator
history   user_id, seq BIGINT, op, message_id     -- the device sync journal
outbound  destination_domain, next_attempt_at, attempt_count, message_id, status
repute    ip_or_domain PK, class, score_1h, score_24h, complaint_rate

• mailbox: one row per user per message, the 124-byte metadata tier saying what this user thinks about the message. msg_seq is a per-user counter that increases with each new message, giving cheap ordering and paging without sorting by timestamp.
• messages: one record per distinct message, shared by all recipients. envelope is the SMTP-level sender and recipient list, distinct from the From: and To: headers a human sees, which can say anything.
• parts: one record per distinct attachment, keyed by the hash of its bytes. blob_locator says which storage bucket and object hold them.
• segments: the per-user search index, in chunks. min_seq/max_seq say which range of messages a chunk covers; dict_offset and postings_locator point into the term dictionary and posting lists (defined in the search deep dive).
• history: a numbered log of every change to a mailbox. An offline device sends the last seq it saw and replays forward.
• outbound: the send queue, one row per message per destination domain, with the next retry time and attempt count.
• repute: reputation scores for the IPs and domains you send from and receive from, over 1-hour and 24-hour windows.

Three decisions rest under everything else:

mailbox is sharded on user_id and nothing else is. Every query a user makes (list a label, page a thread, search, sync a device) is about one user, so one shard answers all of them. messages and parts are keyed by content hash, which has no locality, so they are partitioned by hashing the content hash itself: the lookup-by-key pattern of the key-value store chapter.

blob_ref is a reference, not a foreign key with a count attached. Keeping an exact reference count across two stores sharded on different keys needs a transaction spanning both, and getting it wrong deletes live data. The correct trade is mark-and-sweep with a grace period: a background pass marks which blobs are still referenced, sweeps the rest, and waits long enough that a blob written seconds ago is never mistaken for an orphan.

A user’s search index is rows in a table, not a separate search service. It is stored as segments (chunks of index written once and never edited) addressed by (user_id, segment_id). Rebuilding an index, deleting a message, or closing an account is then a row operation confined to one shard, not a rewrite of a structure shared with everybody.

High-level architecture

The diagram has three flows, best traced one at a time. On the left, top to bottom, is the inbound path from the internet into storage. The middle branch is the read path from clients to the Read API. The bottom branch is the outbound path back to the internet. Cylinders are stored data; rectangles are services.

flowchart TB
    NET(["internet MTAs"]) --> GATE["Connection gate<br/>IP reputation, RBL, rate<br/>rejects before DATA"]
    GATE --> RX["SMTP receiver<br/>parse, DKIM verify"]
    RX --> FILT["Spam + policy<br/>content model"]
    FILT --> DEL["Delivery service<br/>fanout to recipients"]
    DEL --> BLOB[("Blob store<br/>content addressed<br/>hot 3x, cold EC 1.4x")]
    DEL --> MBX[("Mailbox metadata<br/>sharded on user_id")]
    DEL --> IDX["Index builder<br/>per-user segments"]
    IDX --> SEG[("Index segments<br/>co-sharded with mailbox")]
    MBX --> JRN[("History journal<br/>per-user seq")]

    APP(["clients"]) --> API["Read API"]
    API --> MBX
    API --> SEG
    API --> BLOB
    JRN --> API

    APP --> SUB["Submission 587<br/>authN, per-account limit, DKIM sign"]
    SUB --> OQ[["Outbound queue<br/>per destination domain"]]
    OQ --> POOL["MTA pool<br/>segmented IPs by sender class"]
    POOL --> NET
    POOL --> BNC["Bounce + DSN processor"]
    BNC --> OQ

    style BLOB fill:#2d6a4f,color:#fff
    style MBX fill:#1d3557,color:#fff
    style GATE fill:#9d0208,color:#fff
    style POOL fill:#bc6c25,color:#fff

The inbound path, box by box

1. Internet MTAs (other organisations’ mail servers) open a connection.
2. Connection gate. Scores the sending address against your reputation data and public blocklists (historically realtime blackhole lists, RBLs) and hangs up before the message bytes are transferred: the cheapest rejection available.
3. SMTP receiver. Parses what survived and performs a DKIM verify, checking the signature that ties the message to a domain.
4. Spam and policy. Runs a content model over what is left. This is the expensive layer, which is why so much was thrown away before it.
5. Delivery service. Does the fanout: one write to the blob store, one row per recipient in the mailbox metadata.
6. Blob store. Content addressed, hot at three replicas and cold at 1.4x erasure coding.
7. Mailbox metadata. Sharded on user_id.
8. Index builder. Produces per-user index segments co-sharded with the mailbox: a user’s index rows live on the same machine as their mailbox rows, so a search is one machine’s work.
9. History journal. Every change is appended to a per-user numbered log that devices replay to catch up.

The read and outbound paths

On the read side, clients talk to a Read API that reads four things and nothing else: mailbox rows, index segments, blobs, and the journal.

On the send side, clients submit through port 587, which authenticates the account, applies a per-account rate limit, and signs with DKIM. Submissions land in an outbound queue partitioned per destination domain, which feeds the MTA pool whose IP addresses are segmented by sender class. The bounce processor handles failure reports that come back and re-queues or suppresses accordingly. A DSN is a delivery status notification: the machine-readable “this did not arrive” message, which itself arrives as ordinary inbound mail.

The arrow that does not exist is a synchronous one from the outbound pool back to the submitting client. Delivery takes anywhere from milliseconds to four days, so the send API returns “queued” and the real outcome arrives later through the bounce processor.

Deep dive 1: one blob, many mailboxes

Delivery is a fanout over a single immutable object. The bytes are written once, then one small row is written per recipient. The code reads as one sentence: one blob write, N metadata writes, N index appends, and then, only then, the 250. parse_parts, thread_of, and nextval_for stand in for machinery that does not matter here; what matters is the order.

import hashlib

def deliver(blob_store, db, index, raw: bytes, recipients: list[int]) -> None:
    """One blob write, N metadata writes, N index appends.
    Blob first, metadata last -- the mailbox row is the linearization point."""
    message_id = hashlib.sha256(raw).digest()[:16]
    body_hash = hashlib.sha256(raw).hexdigest()

    if not blob_store.exists(body_hash):              # idempotent by construction
        blob_store.put(body_hash, raw)                # a retry rewrites the same bytes

    for part in parse_parts(raw):                     # attachments split out here
        if not blob_store.exists(part.content_hash):
            blob_store.put(part.content_hash, part.data)

    with db.transaction():                            # one shard per recipient batch
        for user_id in recipients:
            db.execute(
                "INSERT INTO mailbox (user_id, msg_seq, message_id, blob_ref, "
                "  thread_id, received_at, flags, labels) "
                "VALUES (%s, nextval_for(%s), %s, %s, %s, now(), 0, %s) "
                "ON CONFLICT (user_id, message_id) DO NOTHING",
                (user_id, user_id, message_id, body_hash,
                 thread_of(raw, user_id), inbox_label()))
    for user_id in recipients:
        index.append(user_id, message_id, terms_of(raw))
    # only now may the SMTP session answer 250

Three properties are load-bearing.

Blob first, metadata last. The mailbox row is the linearization point: the single instant at which the message becomes real for that user. Before it commits, nobody sees the message; after, everybody does. Both orderings have a failure mode, and you choose which to suffer:

• Metadata first, then a crash: a mailbox row points at bytes that are not there, a visible, unfixable defect.
• Blob first, then a crash: an orphaned blob, reclaimed later by the sweep, that no user ever notices.

Writing the blob first makes the survivable failure the one that happens.

ON CONFLICT DO NOTHING on (user_id, message_id). Repeated deliveries are ordinary: if the 250 is lost on the wire, the sending server assumes failure and re-delivers the identical message. Delivery must be idempotent: safe to repeat, producing the same end state. Keying the row on a hash of the message’s own bytes lets the database quietly discard the second insert, the same trick as content-addressed uploads.

The 250 comes after both. Answering earlier turns any crash into permanent data loss, for a message whose sender has already deleted its only other copy on the strength of your 250.

Deletion inverts cleanly: removing the mailbox row is immediate and visible, and the blob survives until no row references it and the sweep runs. Content addressing makes it impossible for one user to delete another user’s copy, by construction and not by a permission check.

Deep dive 2: search, and why the index is per user

Search is the second-largest system you own. Sizing the index from first principles shows that building one per user (paying the fanout a second time) is worth it, on three grounds: query cost, privacy, and deletion.

Vocabulary first, because none of the arithmetic makes sense without it:

• An inverted index maps each word to the list of documents containing it, so a query is a list lookup instead of a scan of every message.
• Each entry in one of those lists is a posting: one “word W appears in document D” fact.
• Delta encoding stores the gaps between consecutive document numbers instead of the numbers. [1004, 1009, 1021] becomes [1004, 5, 12], and small numbers compress into fewer bytes. This is why a posting costs about two bytes instead of six.
• A positional index also records where in each document a word occurred, which makes exact phrase search possible. It stores one entry per occurrence instead of one per distinct word, so it costs several times more.
• Heaps’ law is the empirical rule that the number of distinct words in a text grows roughly as the square root of its length. A message ten times longer has only about three times as many different words, because common words repeat.

How big is an inverted index

The intuition first: an index only lists the distinct words, and longer messages repeat words, so the index grows much slower than the text it covers. Now the arithmetic that pins the fraction down. A 10 KB message is about 1,500 words. By Heaps’ law (10 x sqrt(1500) ≈ 387) that is 387 distinct terms, each a 2-byte posting, so 387 x 2 = 774 bytes of index for 10,000 bytes of text: 7.7%. Add positions and you store one posting per occurrence, 1,500 x 2 = 3,000 bytes, or 30% of the text and 3.9x the index. The common “indexes are 30% of the corpus” figure is the positional case, which this design does not use.

The postings are small because a per-user document number is small: one mailbox holds 54,750 messages over five years, which fits in 16 bits, and delta encoding brings the average below two bytes. A global index would need a document number spanning 16.4 trillion messages (9 B/day for 1,825 days), 44 bits before compression. What a document number refers to is what sets the index’s size, long before any tuning matters.

Scaling one mailbox to the fleet, and adding the term dictionary (the list of distinct words, ~20 bytes per term):

• Postings per user: 54,750 x 774 B ≈ 42 MB.
• Dictionary: ~90,623 distinct terms x 20 B ≈ 1.8 MB.
• Per-user index: ~44 MB, 8.1% of the text it covers (slightly above the per-message 7.7% because of the dictionary, which grows as a square root and so gets relatively cheaper as a mailbox fills).
• Across 300 M active mailboxes: 13.3 PB.

Positional postings would take the fleet index to 49.8 PB: 36.6 PB extra for phrase search. The alternative is to ship non-positional postings and re-scan the top ~100 candidate messages for the phrase directly: 100 x 10 KB ≈ 1 MB of blob reads per phrase query, cheap precisely because the non-positional index already shortened the candidate list.

One number worth holding: the fleet index is measured against 164 PB of indexed text, while the unique messages are only 27.4 PB. They differ by the fanout of 6, because per-user indexing analyses the same message once for every recipient.

Why per user rather than one global index

Three arguments say per user, and only the first is about performance.

Query locality. Take a term appearing in 5% of all mail. Its global posting list over five years is 9 B/day x 1,825 x 0.05 x 2 B ≈ 1.64 TB, of which one user’s share is 54,750 x 0.05 x 2 B ≈ 5,475 bytes: a ratio of 300 million to one, which is exactly the active user count, because one user holds one 300-millionth of the mail. A global index answers “who wrote about invoices” by reading everyone’s postings and filtering by owner; a per-user index reads one 44 MB structure on the shard that already holds the mailbox.

Privacy is a property of the partition, not the query. In a global index, keeping accounts apart means a WHERE owner = me filter applied after retrieval, which a ranking bug, a new code path, or a word-suggestion feature can each leak past. In a per-user index the other accounts’ postings are not in the structure being read at all. Controlling access through the partition survives refactoring, whereas controlling it through a query condition does not.

Deletion is a row operation, not a rewrite. Deleting an account under GDPR drops that user’s segments and is finished. In a global index the same request means rewriting every posting list containing any of that user’s 90,623 distinct words, each scattered through a 1.64 TB structure.

The cost, honestly: you index each message once per recipient, so 164 PB of text is analysed instead of 27, and you lose any signal that spans users. There is no global measure of word rarity (the inverse document frequency, IDF, that ranking leans on) and no “popular in your organisation” suggestion. That 6x is the bill for privacy and query locality.

Index-write throughput

A system can fit on disk and still be unable to keep up with the writes, so size those separately. Write amplification is the ratio of bytes actually written to disk to bytes the application handed over. Log-structured merge (LSM) storage rewrites data as it merges sorted files in the background; a factor of ten is normal.

At 90,000 deliveries/s x 774 B = 69.7 MB/s, times 10x amplification, that is 697 MB/s of index writes across the whole fleet: about 0.7 of one NVMe device (a solid-state disk on the machine’s high-speed bus, roughly 1 GB/s sequential). Indexing is not the throughput problem: the postings are 7.7% of text that had to arrive anyway, and the writes are appends, not scattered updates.

The three index costs that are real: storage (13.3 PB), fanout (774 B per recipient, so a 1,000-recipient list writes 774 KB of index at once), and rebuild time when the segment format changes. Segments are appended per user and merged lazily in the background, a process called compaction. A message becomes searchable the moment its postings land in a small in-memory tip segment, not when compaction finishes.

Deep dive 3: attachments

Attachments are separated from everything else because they dominate the bytes and participate in none of the features. Of the 70,000-byte average message, the attachment part is 0.20 x 300,000 = 60,000 bytes: 85.7% of the bytes, and 0% of them are searched, listed, threaded, or sorted.

That asymmetry drives splitting a message into pieces that live in different places:

The part manifest is the list of attachments belonging to a message, each entry recording content hash, size, media type, and filename. It joins the small metadata row to the large blobs.

Two consequences follow. Drawing a mailbox list touches zero attachment bytes: the interactive path reads only the 1.06% metadata tier, which is why a 200 ms p99 on listing is reachable. And the same attachment sent by twelve people is stored once, which is the 30% duplicate rate worth 27 TB/day.

The filename is not part of the blob. The same PDF arriving as q3.pdf and as Q3 final (2).pdf is one blob with two manifest entries, because the name is something one message says about the bytes while the bytes are global. Putting the name in the identity would defeat the dedup.

Deduplication happens on the server only. A client allowed to ask “do you already have this hash?” before uploading has a membership oracle over everybody’s storage: a way to test whether a specific file exists anywhere. The timing difference between “upload the whole file” and “never mind, you have it” is seconds against milliseconds, so no statistical care is needed to read the answer. The mitigation is to accept the upload anyway, with a randomized threshold, so the client cannot tell. This matters here because email attachments are exactly the guessable documents such an attack targets: contracts, payslips, standard tax forms.

Deep dive 4: SMTP is store-and-forward, so status is eventually consistent

“Was it delivered?” is a question the system genuinely cannot answer. This is not a design choice; it is what the protocol is. Each hop accepts responsibility, answers 250, and takes on the duty to deliver or to report failure later. A 250 from the next hop means “I have it on disk”, not “the human has it”.

SMTP replies are three-digit codes; only the first digit matters here. 2xx is success, 4xx is a temporary refusal (“try again later”), 5xx is a permanent refusal (“never come back”). One outbound message moves through:

flowchart LR
    Q["queued"] --> A["attempt"]
    A -->|"250 accepted"| H["handed off<br/>NOT delivered"]
    A -->|"4xx deferred"| R["retry with backoff"]
    R --> A
    A -->|"5xx rejected"| B["hard bounce · DSN"]
    H -->|"DSN arrives<br/>minutes to days"| B
    H -->|"silence"| U["assumed delivered<br/>unverifiable"]
    R -->|"queue lifetime expires"| B

    style U fill:#bc6c25,color:#fff
    style B fill:#9d0208,color:#fff
    style H fill:#1d3557,color:#fff

A 250 hands the message off but does not deliver it. A 4xx sends it into retry with backoff, each attempt waiting longer than the last. A 5xx is a hard bounce, reported as a DSN. From “handed off”, a DSN may still arrive minutes to days later if the message failed downstream, or silence, in which case the message is assumed delivered, a state that is genuinely unverifiable. A message deferred until the queue lifetime runs out also ends as a bounce.

Retries, sized

The delay doubles from one minute up to a four-hour cap, and the schedule runs until the queue lifetime of 96 hours is used up. The doubling ramp (1+2+4+...+128 min ≈ 4.25 h) is 8 attempts; the remaining ~91.75 hours at the 4-hour cap add ~23 more, for 31 attempts over four days.

Four days is not arbitrary. RFC 5321, which defines SMTP, asks senders to keep trying for at least four to five days, and receivers rely on it. Greylisting is the clearest example: a receiver deliberately answers 4xx to any sender it has not seen, assuming spam software will not come back, so the first message between two correspondents is designed to be delayed by minutes. A sender that gives up after three attempts is not being decisive; it is failing to implement the protocol, and its mail will not arrive.

Queue depth when a receiver goes down

One large provider having a bad afternoon is routine. Take a destination that normally takes 10% of your outbound, down for 4 hours: 15,000/s x 0.10 x 14,400 s = 21.6 M messages, at 70 KB each ≈ 1.51 TB of queue.

That is why the outbound queue is partitioned by destination domain. One slow domain must not cause head-of-line blocking: where the item stuck at the front of a queue holds up everything behind it, regardless of destination. A second reason for per-domain partitioning: the retry state for a domain answering 4xx is a property of that domain, so knowing “gmail is deferring right now” is useful once, for all of it.

Bounces are out of band

The DSN that tells you a message failed arrives as a separate inbound message, minutes to days later, from a server you never spoke to. So delivery status has at least four states (queued, handed off, bounced, expired) and “handed off” looks final without being final. Three consequences:

• No user interface may present the status as a completed result. It mostly moves forward, but it is not settled for four days.
• Bounce processing is a full inbound pipeline in its own right, and it must resist forged DSNs, since anyone can send you a message claiming your mail failed.
• A hard bounce must feed the suppression list: the set of addresses you refuse to send to again. Continuing to mail a non-existent address is the fastest way to destroy your sending reputation.

Deep dive 5: spam is the largest subsystem you own

The 9 billion daily deliveries are what survives filtering, not what arrives. If 60% of attempts are spam, deliveries are the other 40%, so inbound attempts are 9 B / 0.40 = 22.5 B/day, or 225,000/s, 2.5x the delivery rate. Spam is 13.5 B/day; rejecting 80% of it at connect time removes 10.8 B/day, saving 756 TB/day of ingest never read. The 11.7 B/day that reach the content scan are 117,000/s, and at 5 ms of CPU each that is 117,000 x 0.005 = 585 cores busy continuously.

Because the filter is the biggest thing you run, it is layered cheap-to-expensive, each layer throwing away as much as it can before handing on the rest:

The ordering is the design. Rejecting at connection time costs one reputation lookup; rejecting after content analysis costs 70 KB of transfer plus 5 ms of CPU. Moving a rejection earlier is worth roughly four orders of magnitude per message.

SPF, DKIM, and DMARC

These three do different jobs:

• SPF (Sender Policy Framework): a DNS record listing which IP addresses may send mail for a domain. It authorizes the sending machine, which is why it breaks when a message is forwarded: the forwarder’s IP is not on the original domain’s list.
• DKIM (DomainKeys Identified Mail): a cryptographic signature over the message itself. It survives forwarding, because the signature travels with the bytes.
• DMARC (Domain-based Message Authentication, Reporting and Conformance): ties either result to the From address the human actually sees, and publishes what a receiver should do when neither lines up.

The gap DMARC closes is that SPF and DKIM authenticate things the reader never sees (a connecting IP, a signing domain in a header) while the address on screen can say anything.

Deep dive 6: outbound reputation, and why a shared pool is a liability

This is the part of the system you do not control: other companies decide whether your mail arrives, and the decision is made about an IP address that many of your customers share. Receivers score the sending IP and sending domain, and those scores decide inbox, junk, or temporary refusal. Reputation is earned slowly and lost in an afternoon, which is what makes this a design problem and not an operations one.

With 100 outbound IPs carrying 15,000 submissions/s, each IP sends 15,000 / 100 = 150/s, or 12,960,000 legitimate messages a day. Replacing a blocklisted IP is slow: a brand-new address sending at full volume is itself a spam signal, so you warm it by raising volume gradually. Starting at 50/day and doubling, reaching 13 million a day needs 2^18 = 262,144, about 19 days.

So one compromised account’s spam run blocklists an IP that carries 12.96 M messages a day for everyone else, and recovery takes 19 days. That is a blast radius argument (how much of the business one failure takes down), not a capacity one. The pool is not too small; it is too shared. Five mitigations, in the order they matter:

• Segment the pool by sender class. Transactional mail (receipts, password resets, low complaint rate) never shares an IP with bulk marketing or brand-new accounts. A contaminated bulk address then costs you bulk deliverability, not password resets.
• Quarantine new senders in a separate warming pool that cannot contaminate the main one; they graduate on measured complaint rate, not age.
• Limit each account’s outbound rate (see the rate limiter chapter), sized so a single compromised account cannot move a shared address’s 24-hour score before detection fires.
• Sign everything with DKIM at submission, so domain reputation (which you control and which survives IP rotation) carries the weight IP reputation cannot.
• Feed hard bounces and complaint feedback loops into suppression automatically. A complaint feedback loop is a report a large receiver sends when its users mark your mail as spam; continuing to send to addresses that bounce or complain is the strongest negative signal a receiver has.

The architecture’s job is to keep the blast radius of any one bad sender small and the recovery path short.

Consistency: what a user may never see

“Eventually consistent” becomes a specification once it is a list of observations, a verdict on each, and the mechanism enforcing it. The three Never rows are the invariants; everything below is a bounded lag you accept.

The one lag worth engineering away

Search immediately after arrival is the case not to accept, because a user just notified about a message very often searches for it a second later. The fix is a tip segment: the newest postings are kept in a small in-memory piece of the index, which search consults before the on-disk segments, so a message is findable the instant it is listed.

That only works if the tip fits in memory. The last 100 messages per user at 774 bytes each is 77.4 KB/user. Holding one for 1% of daily actives concurrently is 300 M x 0.01 x 77.4 KB ≈ 232 GB across the fleet; holding one for every daily active at once would be 23.2 TB. The first figure is the one that matters, because it is sized by how many people are active at once. Two rules follow: build the tip when a session opens and drop it when the session goes idle, and have search read the tip first and on-disk segments second. Compaction then runs on its own schedule with nobody waiting on it.

Unread counts are the clean counterexample: recomputable from the mailbox rows, so a cache and never a source of truth. A count that drifts is repaired by a recount; a message that drifts is gone.

Bottlenecks, scaling, and failure modes

Every limit derived so far, with the mechanism that keeps it from becoming an outage.

The blob-write line is the clearest single statement of what dedup buys. Naive storage writes at the delivery rate; deduplicated storage writes at the submission rate, because the other five copies are already there: 90,000 x 70 KB x 8 = 50.4 Gbps becomes 15,000 x 70 KB x 8 = 8.4 Gbps. At 1 Gbps of sustained write per storage machine, that is the difference between ~51 machines absorbing writes and ~9, all from one decision about what a stored object is named after.

Failure modes

Each row is a concrete failure, the signal that detects it, and the design property that contains it. The first row is the only one with no detection, which is why the 250 ordering is non-negotiable.

Alternatives rejected

Each entry names a plausible alternative, what it is good at, the number that rules it out here, and where it would instead be right.

One copy per recipient. Good: deletion is trivial, no reference counting, no sweep. Rejected on 1.15 EB against 142 PB: an 8x storage bill to avoid writing a garbage collector. Correct below the point where storage costs less than the engineering, which for a small tenant-isolated mail product is a real place to be.

One global inverted index. Good: relevance signals that span users, a global measure of word rarity, one index to operate instead of 300 million. Rejected on both axes: a single ordinary term’s posting list is 1.64 TB of which a querying user needs 5,475 bytes (300 million to one), and cross-account isolation becomes a query predicate that one bad code path removes. Correct for a corpus that is genuinely public, such as web search, where there is no owner to isolate.

Positional postings by default. Good: exact phrase search with no second pass. Rejected on 30% of the text instead of 7.7%: 49.8 PB of index instead of 13.3, so 36.6 PB of extra storage for a minority query type. Non-positional plus a re-scan of the top candidates gets the same answer for about a megabyte of blob reads.

Relational storage for message bodies. Good: one system, transactional consistency between body and metadata. Rejected because bodies are immutable blobs with no predicates over them. The database would hold 142 PB to serve queries that only touch the 2 PB of metadata. The split is not a scaling workaround; the two halves have different access patterns and consistency needs.

Synchronous delivery confirmation to the sender. Good: the sending user learns the truth. Rejected because the truth takes up to four days by protocol design, and holding a request open for four days is not an API. The honest design returns “queued”, delivers the outcome asynchronously, and refuses to render “handed off” as “delivered”.

One shared outbound IP pool. Good: simpler, better per-IP volume, faster warm-up. Rejected on blast radius: one compromised account can blocklist an IP carrying 12.96 M legitimate messages a day, and the replacement takes 19 days to warm. Segmentation costs IPs and warm-up time; it buys a bounded failure domain, and reputation failures are the ones you cannot engineer your way out of after the fact.

Per-user encryption keys at rest. Good: a stolen blob store is useless to the thief, and a compromised operator cannot read mail. Rejected for the dedup path: encrypting the same plaintext under two keys produces two different ciphertexts, so nothing matches anything, and the corpus reverts from 142 PB to 1.15 EB, over 1,000 PB of extra storage. Convergent encryption (deriving the key from a hash of the plaintext, so identical content encrypts identically) restores dedup but reintroduces the membership-oracle attack: anyone who can guess the contents can encrypt their guess and confirm the ciphertext already exists. The choice is between ~1,000 PB of extra storage and an oracle that confirms guesses. Make it deliberately, not during a security review.

Conclusion

The load-bearing ideas, in the order the chapter builds them:

• Fanout is 6, and it decides everything. Every subsystem either pays it or dodges it. Storage dodges it (one blob, six references, 83% saved). Metadata pays it cheaply (124-byte rows, 1% of writes). The search index pays it deliberately (164 PB of text analysed instead of 27) to buy privacy and query locality.
• Email is a storage problem behind a delivery protocol. Dedup plus tiering turns 3.45 EB into 203 PB, a 17x reduction. The two halves (immutable blobs and mutable per-user metadata) have different access patterns and different consistency needs.
• The search index is per user and stored as rows. A per-user document number keeps a posting at 2 bytes; isolation becomes a partition property, not a WHERE clause; deletion drops a segment.
• SMTP is store-and-forward, so status is eventually consistent by protocol. A 250 means “on my disk”, never “delivered”. Durability must precede the 250, and no UI may render “handed off” as “delivered”.
• Deliverability is a reputation system run by other people. The design’s job is to keep the blast radius of any one bad sender small (segment IP pools, rate-limit accounts, sign with DKIM) and the recovery path short.

Key numbers

Further reading

• RFC 5321, Simple Mail Transfer Protocol: the store-and-forward model, reply codes, and the four-to-five-day retry expectation.
• SPF, DKIM, and DMARC, the sender authentication trio covered above: a DNS record authorizing sending IPs, a cryptographic signature over the message, and the policy that aligns either result to the visible From address.
• The database internals chapter: the log-structured merge storage behind the index write-amplification figure.
• The file storage chapter: content addressing and the dedup side channel this chapter reuses; the notification system chapter reaches the same “delivered is a lie” conclusion from the push side; the message queue chapter is the outbound queue; the hotel reservation chapter is the same interview question asked where the invariant, not the storage, is the constraint.

One line to remember: store the body once and pay the fanout of 6 only where it buys you something, never on bytes, deliberately on the search index.