How to design a notification system

A fan-out engine whose last hop crosses into a third party you do not own, and that one fact generates every hard problem: no exactly-once, cost-driven routing, latency-split queues, a consent gate, and tracking you cannot trust.

Two facts that shape everything

• No exactly-once. No transaction spans your DB and the provider’s (APNs/FCM, Twilio, SMTP). Build at-least-once (retry until acked, accept duplicates) plus a client dedup key in the payload.
• Channels differ ~4 orders of magnitude in cost, so routing is a P&L decision before a technical one.
• Input (user, category, template, params) → a message on a device, in the user’s language, subject to consent, with an event trail.

The numbers

• Volume: 100 M DAU × 5 = 500 M/day = 5,000/s avg, 15,000/s peak (3x). Mix 80/15/5 → 400 M push, 75 M email, 25 M SMS.
• Latency: transactional p99 < 30 s; marketing best-effort (~1,000x gap) → cannot share a queue.
• Rounding convention: 86,400 s/day → 1e5 (costs ~16%).

• SMS = 5% of volume, 96% of bill. SMS/push = 20,833x; email/push = 278x.
• Push fleet: 6 boxes send (6 × 2,000/s HTTP/2 = 12,000/s cap), ×5 for support = 30 boxes = cost number, never throughput. Honest sizing at 0.8 util = 8 senders.

Queue: isolation, not throughput

• Synchronous fan-out is impossible: 10 M-follower post = 10M / 12,000 = 833 s. Fan-out must be async behind a queue.
• Little’s Law: in-flight = rate × latency. 15,000/s at 50 ms = 750 slots; at 2 s = 30,000 (40x jump). Fixed pool → blocking → slow APNs stalls email and SMS.
• Queue turns the mismatch into backpressure (bounded, visible backlog): 11,500/s accrual = 6.9 M msgs = 3.45 GB per 10 min.
• Queue must be durable/off-heap, have TTL + drop policy, and be partitioned by latency class. Every queue needs a dead-letter queue for poison messages.
• Per-category TTL: OTP 60 s · ride 300 s · order_shipped 86,400 s · marketing 3,600 s.

Consent gate + fatigue (the expensive part)

• Gate (opt-out, quiet hours, per-user budget) runs in the worker, last ms before the send — never at fan-out (stale consent). Suppressed = dropped and recorded (distinct from failed).
• Fatigue = each extra message raises permanent opt-out odds. 100M × 0.40 × 2 × 0.0005 = 40,000 opt-outs/day → × 365 × $5 = $73 M/yr, 5.3x the SMS saving, on no dashboard. The 0.0005 needs a holdout to measure.
• Budgets: transactional none · social 3/day · marketing 2/week. Unknown category falls to strictest (marketing), never uncapped, never its own bucket.
• Quiet hours stack load: one timezone boundary release = 1,041,667/s (69x peak). Fix: quiet_end + uniform(0, W); W = 1 h lands exactly on peak, ship W = 4 h. Store each user’s timezone.
• Opt-out bug = TCPA statutory damages, $500/message: 0.01% leak on 25 M SMS = 2,500 = $1.25 M/day. STOP/UNSUBSCRIBE writes to preferences directly; email unsubscribe is one-click GET + List-Unsubscribe.

Routing: the fallback ladder

• Try cheapest channel that can reach the user; escalate only on evidence (no receipt in 5 min) and by category, never by preference. OTP → SMS; “liked your post” never escalates.
• Push if token seen in 30 days. 20% SMS deflection = 5 M/day off SMS = $13.7 M/yr ($684k/yr per point); instrument the deflection rate.

Exactly-once is not available

• Two steps (call provider, record locally) in different failure domains, no 2PC, no query API. APNs apns-collapse-id collapses (loses) messages; FCM has no idempotency key.
• Order is your only choice: commit-then-send = at-most-once (silent loss); send-then-commit = at-least-once (visible duplicate). Choose send-then-commit; record unknown after a timeout, never failed.
• Duplicate rate ~1 in 2,000; 97% from ambiguous timeouts (250k/day), 3% from crashes (7,500/day) — tightening the crash window buys almost nothing.
• Make it invisible: client keeps 1,000 ids × 8 bytes = 8 KB, drops repeats (why notification_id is an 8-byte Snowflake, not 16-byte UUID). Server dedup in Redis, TTL = 4x retry ladder (3,600 s) = 1.2 GB; only collapses your own retries. Claim = exactly-once as observed by the user.

send ──▶ provider ──▶ device (delivered)
   ◀─x── ack lost (timeout): outcome UNKNOWN
retry ──▶ provider ──▶ device (delivered AGAIN)
                          └─ client dedup on id drops repeat

Retries

• Exponential backoff, full jitter uniform(0, min(cap, base×2^n)); honor Retry-After but never sleep exactly it (rebuilds the herd). Retry at one layer only (3 tiers × 3 = 27 calls). Cap a retry budget as a fraction of successes (10% → 1.1x load, not 4x).
• Classify before scheduling: Unregistered/InvalidRegistration → reap token, never retry; PayloadTooLarge → fix template, never retry; 5xx/timeout → retry.
• Retry ladder must fit inside the TTL: OTP 3 attempts in 60 s; order_shipped 6 in 15 min.

Templates + SMS trap

• Corpus 500 × 40 locales × 2 KB = 40 MB, in-process (no RPC). Render late (send time): queue carries 60 GB refs vs 250 GB strings, and a template fix still reaches the backlog.
• Locale belongs to the user not the request; use ICU MessageFormat for plurals/gender; fallback pt-BR → pt → en; validate params_schema at POST (422, not “Hi ,”).
• SMS segment = billing unit. GSM-7 = 160 chars/segment; one non-GSM-7 char drops the whole message to UCS-2 = 70 (67 concatenated). 150-char + 1 emoji = 3 segments. 10% of 25 M with 2 extra segments = $37,500/day. Enforce a segment-count lint rule in CI using the full GSM 03.38 table (£ ¥ § ¡ ¿, accented Latin, € in extension) — ASCII-only rejects valid German/French.

Tracking: “delivered” is a lie

• Apple MPP pre-fetches images → every Apple recipient = an open. 55% Apple + 45% at 25% true = 66.25% reported (2.65x inflated), and it tracks device share. Report the non-Apple cohort + coverage, or don’t report opens.
• Metric that works: ask the device for a rendered receipt (rides the dedup channel, free) = 320 M/day, part of 15,000 writes/s (3 events/notification, 3x send rate). Store in a separate columnar analytics DB, never synchronous on the send path (27.4 TB/yr). Always publish coverage next to the rate.
• Reap dead tokens on 410 Unregistered/NotRegistered in the same code path. 1% monthly uninstall → 1 - 0.99^12 = 11.4% dead after a year = 45.6 M/day to nowhere; every rate vs sent is 11% wrong.

Data model gotchas

• device_tokens shard by user_id (fan-out reads all a user’s devices = one hop), not device_id (scatter-gather).
• preferences keyed (user, channel, category) for a point lookup on the hot path, not a JSON blob.
• outbox row per (notification, channel) — the ladder needs per-channel state/TTL/attempts.
• Scaling axis that matters is queue partition count, not sender throughput.