How to design an email service

An email service is a storage-and-retrieval problem behind a delivery protocol: a message is submitted once but delivered to many mailboxes, so the body is stored once while the search index is built once per recipient.

The one decision: where the fanout lands

• Fanout = mailboxes one submission is delivered to; here 9 B delivered / 1.5 B submitted = 6.
• Storage dodges it: one blob (opaque lump of bytes) with N references. Body stored once.
• Search index pays it deliberately (per user) to buy privacy and query locality.
• Metadata pays it cheaply: a 124-byte mailbox row per recipient, ~1% of writes, 100% of what a user can change.

submit once ──► 1 blob (body, immutable, shared)
             └► N mailbox rows (per user, mutable, sharded on user_id)
             └► N index appends (per user segments)
             then ──► SMTP 250

Sizing (back of envelope)

• Assumptions: 1 B mailboxes, 300 M DAU, 30 in / 5 out per day, 10 KB text, 20% carry 300 KB attachment, peak = 3x.
• Deliveries: 300M x 30 = 9 B/day = 90,000 QPS (270k peak). Submissions 1.5 B/day = 15,000 QPS.
• Bytes/message (weighted avg): 10,000 + 0.20 x 300,000 = 70,000 B; attachment = 85.7% of bytes.
• Storage: 630 TB/day naive (1.15 EB / 5 yr) → 78 TB/day deduped (142 PB). Message dedup saves 5/6 = 83.3%.
• Physical: hot (30 days, 3 replicas) + cold (erasure coded 1.4x) ≈ 203 PB vs 3.45 EB naive = 17x (8.08x dedup × 2.10x tiering).
• 1 PB = a million GB; 1 EB = a thousand PB.

Two-tier design

• Blob tier: immutable, content-addressed (named by hash of its bytes), eventually consistent, cheap. Identical content shares one copy with no coordination.
• Metadata tier (mailbox): mutable, strongly consistent, sharded on user_id so every per-user query hits one machine.
• Reclaim via mark-and-sweep with grace period, not transactional reference counts. blob_ref is a reference, not a foreign key.
• Delete = drop the mailbox row (immediate); blob survives until no row references it. Content addressing means one user cannot delete another’s copy.

Search: per user, stored as rows

• Inverted index: word → list of docs. Each posting ≈ 2 bytes via delta encoding (store gaps, not doc numbers). Small per-user doc numbers (54,750 msgs fit in 16 bits) keep postings tiny.
• Heaps’ law: distinct words ≈ √(length). A 10 KB (~1,500 word) message → ~387 distinct terms → 387 x 2 = 774 B = 7.7% of text.
• Per-user index ~44 MB (8.1% with dictionary); fleet 13.3 PB.
• Positional index (phrase search) = one posting per occurrence = 30% of text, 3.9x. Skip it: re-scan top ~100 candidates (~1 MB of blob reads).
• Why per user, not global: query locality (one 44 MB read vs 1.64 TB list), privacy is a partition property not a WHERE filter, deletion drops a segment (GDPR). Cost: 164 PB analysed vs 27, no cross-user signal (no IDF).

SMTP is store-and-forward → status is eventually consistent

• 250 = “on my disk”, NOT “delivered”. Once returned, sender deletes its copy, so durability MUST precede it.
• Reply codes: 2xx success, 4xx temporary (“retry”), 5xx permanent (“hard bounce”).
• Retries: delay doubles 1 min → 4 h cap, over 96 h queue lifetime = 31 attempts / 4 days. RFC 5321 asks 4–5 days; greylisting deliberately 4xxs first contact.
• Delivery outcome arrives out of band as a DSN (delivery status notification), a separate inbound message. States: queued, handed off, bounced, expired — “handed off” looks final but is not.
• Hard bounce → suppression list; mailing dead addresses destroys reputation.

Deliverability = reputation run by other people

• 100 outbound IPs, 15,000 subs/s → each IP carries 12.96 M msgs/day. Warming a fresh IP (50/day doubling) ≈ 19 days.
• One compromised account can blocklist a shared IP → blast radius, not a capacity problem. The pool is too shared, not too small.
• Mitigations: segment pools by sender class; quarantine new senders; per-account rate limits; sign everything with DKIM; feed bounces/complaints to suppression.

Sender authentication

Gotchas

• Blob first, metadata last. The mailbox row is the linearization point. Blob-then-crash = harmless orphan; metadata-then-crash = row pointing at missing bytes (unfixable). Never the second ordering.
• 250 only after blob AND rows commit — no detection exists if you answer early.
• Delivery is idempotent: ON CONFLICT (user_id, message_id) DO NOTHING; a lost 250 triggers identical re-delivery.
• Search API takes no user param — caller identity selects the index; partition, don’t filter.
• Attachments fetched by content hash, not message id; filename lives in the part manifest, not the blob (putting it in the identity defeats dedup).
• Server-only dedup: a client “do you have this hash?” check is a membership oracle; accept the upload anyway with a randomized threshold.
• Spam is the largest subsystem: 60% spam → 22.5 B attempts/day (225k/s, 2.5x delivery). Reject cheap-to-expensive (connection gate → envelope → auth → content scan = 585 cores); moving a rejection earlier is worth ~4 orders of magnitude.
• Tip segment: newest postings kept in memory so a just-arrived message is searchable instantly; compaction runs async behind it.