How to design a notification system

In this lesson, we’ll design the service every other service calls when it needs to reach a person: an order ships, a login code is issued, someone likes a post. Each of those has to become a message on a device. By the end you’ll be able to size the sender fleet from a workload, name the two facts that force the whole architecture, and defend the queue layout, the consent gate, the retry rules, and the tracking pipeline in an interview.

We build the whole thing on two facts:

1. Guaranteed once-and-only-once delivery is not achievable here.
2. The three delivery channels differ so much in unit cost that choosing between them is a financial decision before it is a technical one.

Everything else follows from those two facts: the queue layout, the consent gate, the retry rules, the tracking pipeline. We’ll take each in turn and watch a specific number force it.

What goes in, and what comes out. The input is a request from an internal service in the shape (user, category, template, params): “user 8891, category order_shipped, template ship_v3, with the tracking number filled in.” The output is a message on a device (a push notification, an SMS text, or an email) in the user’s language, subject to that user’s consent, with a trail of events recording what the system attempted and what came back. Everything in the middle is fan-out, filtering, and queueing.

Two terms used throughout:

• Fan-out turns one logical request into many concrete deliveries. One “your team posted” becomes 10 million individual pushes; one notification becomes an attempt on each of a user’s devices.
• A channel is one delivery road: push, SMS, or email.

The two facts that shape everything

The architecture is a queue and some workers, which is easy to draw. What matters is being precise about delivery semantics and cost.

Exactly-once delivery is not available

Exactly-once delivery means every notification arrives, and arrives exactly once: never zero times, never twice. You cannot have it here, because every hop that matters crosses into a third party you do not own:

• APNs (Apple Push Notification service) and FCM (Firebase Cloud Messaging) deliver to phones.
• Twilio or a similar aggregator delivers SMS.
• An SMTP relay delivers email. SMTP is the protocol mail servers speak to each other.

No transaction, an all-or-nothing bundle of operations, spans your database and theirs. Without one, “the message went out” and “I recorded that it went out” cannot succeed or fail together, and that gap is the whole problem.

What you build instead is at-least-once delivery: keep trying until the other side acknowledges, and accept that some messages arrive twice. You then add a dedup key carried inside the payload so the receiver can recognize a repeat and discard it. That is the honest design; claiming exactly-once is not.

The channels differ by four orders of magnitude in cost

SMS turns out to be about 5% of volume and 96% of the bill, so routing between channels is a financial decision. The back-of-the-envelope section prices this out.

Requirements

Functional

• Accept a request from any internal service: (user, category, template, params).
• Deliver over push (iOS/Android), SMS, and email, with a fallback ladder between them.
• Per-user preferences: channel opt-in, category opt-in, quiet hours, locale.
• Templates with localization and versioning.
• Scheduled and immediate sends; a bulk send to an audience segment.
• Track what happened to each notification, and expose it.

Three terms there need glossing:

• A fallback ladder is an ordered escalation: try the cheapest channel that can reach the user, and move to a costlier one only on evidence that the first did not work.
• Quiet hours are a per-user window, in the user’s own timezone, during which nothing non-urgent may be sent.
• An audience segment is a saved set of users a campaign targets, such as “everyone who abandoned a cart this week.” (SMS billing uses the word segment for something else entirely, covered later; keep the two apart.)

Out of scope: rendering the in-app inbox, computing who is in a segment, and the campaign authoring interface.

Non-functional

Each row below is a number that forces a structural choice later. The two latency rows matter most, because their targets differ by a factor of about 1,000.

p99 under 30 s is the 99th percentile of end-to-end latency: 99% of notifications finish faster than that. Stating a percentile, not an average, makes a promise about the slow tail, which is what users notice.

Transactional traffic is something the user’s own action asked for: a login code, a receipt. Marketing traffic is something you decided to send. This split runs through the whole design. Two latency classes with a 1,000x difference in tolerance cannot share a queue, because the marketing backlog would sit in front of the OTP. That single fact drives the queue layout.

An OTP is a one-time passcode: the short login code valid for about a minute.

The assumptions the numbers rest on

Every figure below is downstream of a short list of assumptions. The ones marked load-bearing pick the architecture: overturn one and you redraw something. The rest only scale a number.

(B8: peak is 3x average. B9: 0.1% of provider calls end ambiguously, half of those delivered. B10: 80% of pushes produce a client receipt; 1% monthly uninstall rate. These scale numbers but change no decision.)

A basis point is one hundredth of a percent, so B6’s 5 basis points is 0.0005: five extra opt-outs per ten thousand users per extra notification. B11 and B12 carry the two largest dollar figures and neither is measured anywhere in this design, so treat those figures as orders of magnitude. What survives the uncertainty is the ordering: fatigue beats the SMS optimization across essentially every plausible setting, and that ordering is what changes the design.

Back of the envelope

We start from one product number, 100 million daily active users, and it yields the request rate, the channel mix, the bill, and the size of the tracking pipeline. Divisions below use the rounding convention 86,400 s/day → 1e5, which turns per-day-to-per-second into a decimal shift and costs about 16%, well inside the error bars here.

Volume. 100 M DAU × 5 notifications = 500 M/day, which is 5,000/s average and 15,000/s at 3x peak. Split by the 80/15/5 mix: 400 M push, 75 M email, 25 M SMS per day.

What each channel costs

Price one million messages on each channel and the spread is four orders of magnitude.

Push has no provider fee (APNs and FCM charge nothing) so its cost is the sender fleet, the only channel whose unit cost you derive instead of look up. At the 12,000/s peak push rate (80% of 15,000) and ~2,000 pushes/s per box over HTTP/2 (which multiplexes many requests over one connection), you need 6 sender boxes. Multiply by 5 for the surrounding infrastructure (the token store, the retry queue, the receipt pipeline) for ~30 boxes at $0.20/hr, which is $144/day, or $0.36 per million pushes.

Two facts about that block are load-bearing later:

• 6 of those 30 boxes send; the other 24 do not. The fleet’s push capacity is 6 × 2,000 = 12,000/s and stays there however large the support tier grows. The 30 is a cost number, never a throughput number. (Dividing by the wrong one changes a later answer by 5x.)
• The 6 boxes were solved for exactly the 12,000/s peak, so utilization is 1.00: zero headroom by construction. Size instead as offered load → chosen utilization → capacity: 12,000 / 0.8 = 15,000/s, which at 2,000/s per box is 8 sender boxes, not 6. The rest of this lesson keeps the 6-box figure because that is what it costs and fans out against, but 8 is the honest sizing.

Email and SMS carry a provider fee you look up: email at $0.10/1,000 is $100 per million; SMS at $0.0075/message is $7,500 per million. Multiply each channel’s per-million price by its daily volume in millions:

That is $195,144/day, $71.2 M/year, and SMS is 96% of it (187,500 / 195,144) while being 5% of the messages. Per message, SMS is 20,833x push and 75x email; email is 278x push.

One trap in those ratios: count push as literally free and the ratio is infinite and useless; count only the provider fee and push and email look identical (both zero at the provider). The honest comparison prices push loaded with the fleet that carries it: $0.36 per million against SMS at $7,500, which is 20,833x. The number that changes the design is none of the ratios, though. It is the 96%.

Storage and the tracking load

The system keeps two data sets, and the second is a system in its own right:

• Token store. A device token is the opaque string APNs or FCM gives an app installation, and the only address a push can reach. At two devices per user, that is 200 M tokens at ~200 bytes each = 40 GB: it fits in the memory of one box, so device lookup is never the bottleneck.

• Event trail. Every notification produces three events: queued, sent, and client receipt. That is 1.5 B rows/day at ~50 bytes = 75 GB/day, or 27.4 TB/year. As a write rate it is 15,000 event writes/s, 3x the 5,000/s send rate. Telemetry here is not a side-channel hanging off the send path; it is a larger system than the send path, and it must not share a database with it. (Push receipts alone, at 80% of 400 M pushes, are 320 M/day and 3,200 writes/s.)

API sketch

The interface internal services call is one endpoint in full plus four in outline:

POST /v1/notifications
  {"user_id": ..., "category": "order_shipped", "template_id": "ship_v3",
   "params": {...}, "channels": ["push","email"], "ttl_s": 86400,
   "idempotency_key": "order-8891-shipped"}
  202 {"notification_id": "..."}      <- accepted, not delivered
  429 over the caller's quota
  422 unknown template, or params fail the template's schema

POST /v1/notifications/bulk        {"segment_id", ...}  -> job handle
GET  /v1/notifications/{id}        -> per-channel attempt history
GET/PUT /v1/users/{id}/preferences -> channels, categories, quiet hours, locale
POST /v1/receipts                  {"notification_id","event":"rendered"}

Four choices there are deliberate:

• 202, never 200. 200 means “done”; 202 means “durably queued,” the strongest true statement available, because nothing has been handed to a provider yet. Returning a status that implies delivery is the API-level version of the exactly-once lie.
• idempotency_key is supplied by the caller and scoped to it. An idempotency key is a caller-chosen identifier for one logical intent (order-8891-shipped); the server records it and, on seeing it again, returns the original outcome instead of acting twice. This is the only class of duplicate you can eliminate: the same upstream event submitted twice. It does nothing about duplicates the provider creates.
• ttl_s is required. A notification is perishable. “Your ride is outside” delivered 40 minutes late is worse than nothing, so the consumer must be able to drop it.
• category, not just template_id. Rate limits, quiet hours, and opt-outs all key on category. A system whose only unit is a template cannot express “never suppress an OTP.”

Data model

Five tables hold everything the system knows.

device_tokens                             -- 40 GB, shard by user_id
  user_id BIGINT, device_id TEXT, platform SMALLINT, token TEXT
  locale TEXT, tz TEXT, app_version TEXT
  last_seen_at TIMESTAMP, invalid_at TIMESTAMP
  PRIMARY KEY (user_id, device_id)

preferences
  user_id BIGINT, channel SMALLINT, category TEXT
  allowed BOOL, quiet_start SMALLINT, quiet_end SMALLINT
  PRIMARY KEY (user_id, channel, category)

templates
  template_id TEXT, version INT, locale TEXT, channel SMALLINT
  body TEXT, params_schema JSONB
  PRIMARY KEY (template_id, version, locale, channel)

outbox                                    -- one row per (notification, channel)
  notification_id BIGINT, channel SMALLINT, user_id BIGINT
  template_id TEXT, params JSONB, ttl_at TIMESTAMP
  state SMALLINT, attempts SMALLINT, last_provider_status TEXT
  PRIMARY KEY (notification_id, channel)

events                                    -- append-only, columnar, 27.4 TB/yr
  notification_id, channel, event, at, provider_status

Three decisions here are worth defending:

• Shard device_tokens by user_id, not device_id. To shard is to split a table across machines that each hold a disjoint slice; the shard key decides which machine a row lands on (see the consistent-hashing chapter). Fan-out reads every device a user owns. With a user_id key that is one query on one hop; with a device_id key it becomes a scatter-gather across the fleet, on the hottest read in the system.

• notification_id is a Snowflake ID, not a UUID (see the ID-generator chapter). A Snowflake ID packs a timestamp, machine number, and counter into 64 bits (8 bytes) and sorts by time; a UUID is 16 random-ish bytes that sort by nothing. This id is also the client’s dedup key, so it travels inside every payload, and the whole push payload is capped at 4 KB. Eight bytes beats sixteen.

• preferences is keyed on (user, channel, category), not one JSON blob per user. The opt-out check is a point lookup on the hottest path in the system, immediately before every provider call. A JSON blob would force the worker to fetch, parse, and re-serialize a document to answer a yes/no question. The outbox row is per (notification, channel), not per notification, because a fallback ladder produces several attempts across channels, each needing its own state, TTL, and attempt count.

High-level architecture

The diagram below follows one notification through the pipeline, with the request entering at the top, the providers at the bottom, and telemetry running down the right. The three parallel queues sit in the middle, and the gate below them is where consent is checked.

flowchart TD
    SVC["Calling services<br/>orders, social, auth"] --> API["Notification API<br/>validate, idempotency key"]
    API --> FAN["Fanout service<br/>resolve users and devices"]
    FAN --> PREF[("Preferences<br/>+ device tokens")]
    FAN --> QT[["Transactional queue<br/>(p99 under 30 s)"]]
    FAN --> QS[["Social queue"]]
    FAN --> QM[["Marketing queue<br/>(best effort)"]]
    QT & QS & QM --> W["Channel workers"]
    W --> GATE{"Consent gate:<br/>opt-out, quiet hours,<br/>per-user budget<br/>checked HERE, not earlier"}
    GATE -->|"suppressed"| DROP["Drop and record"]
    GATE -->|"allowed"| TPL["Render template<br/>locale, ICU plurals"]
    TPL --> APNS["APNs / FCM"] & SMS["SMS aggregator"] & MAIL["SMTP / ESP"]
    APNS & SMS & MAIL --> EV[["Event stream"]]
    CLIENT["Device"] -->|"rendered receipt"| EV
    EV --> ANA[("Analytics store<br/>columnar")]

    style GATE fill:#9d0208,color:#fff
    style QT fill:#2d6a4f,color:#fff
    style QS fill:#1d3557,color:#fff
    style QM fill:#bc6c25,color:#fff

• Into the API. Calling services post to the notification API, which validates the request and records the idempotency key.
• Into a queue. The fan-out service expands one request into concrete deliveries by resolving the user’s devices and preferences, then drops each into the transactional, social, or marketing queue by how urgent the category is. Channel workers drain the queues.
• Through the gate. Each worker’s first act is the gate (opt-out, quiet hours, per-user budget), checked here and nowhere earlier. A notification that fails is marked suppressed, then dropped and recorded, because “we chose not to send this” and “we failed to send this” must never look the same on a dashboard.
• Rendered. A notification that passes is rendered from its template in the user’s locale, with correct plural forms (ICU plurals).
• Handed to a provider. APNs or FCM for push, an SMS aggregator for text, SMTP or an ESP (email service provider, such as SendGrid) for email.
• Recorded. Every provider response, plus every rendered receipt the device posts back, lands on the event stream and settles in a columnar analytics store: one that keeps each field’s values together, not each row’s, which is what makes counting over billions of rows cheap.

Three claims the picture makes, each expanded below: queues split by latency class, not channel; the consent gate lives in the worker immediately before the provider call, not in the fan-out service; and the client feeds the event stream, because providers will not tell you the truth about delivery.

Deep dive 1: cost drives routing

The fallback ladder is what the cost table forces. Try the cheapest channel that can reach the user, and escalate only on evidence the cheap one did not work.

flowchart LR
    N["Notification"] --> P{"Live push token<br/>seen in 30 days?"}
    P -->|"yes"| PUSH["Push<br/>$0.36 / M"]
    PUSH --> R{"Client receipt<br/>within 5 min?"}
    R -->|"yes"| DONE["Done"]
    R -->|"no"| E{"Category allows<br/>escalation?"}
    P -->|"no"| E
    E -->|"yes, and email opted in"| MAIL["Email<br/>$100 / M"]
    E -->|"transactional only"| SMSN["SMS<br/>$7,500 / M"]
    E -->|"no"| DROP["Stop"]

    style PUSH fill:#2d6a4f,color:#fff
    style SMSN fill:#9d0208,color:#fff

If a push token was seen in the last 30 days, push, because it is nearly free. If a client receipt comes back within five minutes, stop: it landed. Otherwise, if the category allows escalation, go to email when the user has email opted in, or to SMS for transactional categories only.

What the ladder saves. Deflection means a message that would have gone by SMS being satisfied by a cheaper channel. At 20% of SMS volume (B11), that is 5 M messages/day off SMS, worth 5M × $7,500/M = $37,500/day, or $13.7 M/year. The saving is exactly linear in the deflection rate ($684,000/year per percentage point), which also tells you what to instrument first: the deflection rate itself. That is why the “seen in 30 days” freshness check and the five-minute receipt window are business logic, not plumbing.

Why the ladder is ordered that way. Six properties differ across the channels:

• An MTA (mail transfer agent) is the server that accepts and forwards email. “MTA acceptance” means the receiving server took the message, which is entirely compatible with filing it straight into spam.
• A DLR is a delivery receipt returned by a mobile carrier; the tracking section shows why it often means nothing.
• GSM-7 is the 7-bit alphabet SMS was designed around; the 160-versus-70 character split it creates is a real line item, derived in the SMS-encoding section.

Push is 20,833x cheaper and strictly less reliable: that is the whole tension. The rule that falls out: escalate on category, never on channel preference alone. An OTP escalates to SMS, because a failed login costs the business far more than three quarters of a cent. A “someone liked your post” never escalates, at any deflection rate.

Deep dive 2: fan-out, and what the queue is for

The queue’s real job is not throughput; it is isolation between channels.

Why a synchronous fan-out is impossible. A synchronous fan-out runs inside the caller’s HTTP request while the caller waits. Take the worst realistic case: one account with 10 million followers posts. The push fleet sends at its real capacity of 12,000/s (6 senders × 2,000, not the 30-box cost figure), so the fan-out takes 10M / 12,000 = 833 s, nearly 14 minutes. (The reflex answer divides by 30 and gets 167 s; that multiplies the whole fleet’s box count by a per-sender rate, and 24 of those boxes never open a connection to APNs.) No HTTP request lives for 833 seconds, so fan-out must be asynchronous behind a queue.

What the queue prevents. Little’s Law: work in flight equals arrival rate times latency, in-flight = rate × latency (derived in the scaling-up chapter). In-flight is exactly the number of concurrent slots you must have. At the 15,000/s peak, a healthy 50 ms provider needs 750 slots; a degraded 2 s provider needs 30,000, a 40x jump, instantly. A fixed thread pool cannot meet that, so the demand turns into blocking: threads sitting idle waiting for Apple. And those threads are shared with every other channel. Without a queue, a slow APNs takes down email and SMS, which have nothing to do with APNs. That coupling is the real argument for the queue.

What the queue turns the outage into. With a queue the mismatch becomes a backlog, not an outage. This is backpressure: excess work accumulates in a bounded, visible buffer instead of failures propagating upstream. Size it: push arrives at 12,000/s while a 1,000-slot pool at 2 s each drains only 500/s, so it accumulates at 11,500/s: 6.9 million messages, 3.45 GB, after ten minutes of one slow provider. Three queue properties follow:

• Durable and off-heap. Off-heap means written to disk in a separate process. An in-process buffer dies with the process and takes all 6.9 million messages with it, a worse incident than the one that created the backlog.
• A TTL and a drop policy. Ten-minute-stale notifications are mostly worthless, and dumping all 6.9 million the instant the provider recovers is a second incident.
• Partitioned by latency class. Otherwise the 6.9 million marketing messages sit in front of the next OTP.

The drop policy is a per-category product decision:

Every queue also needs a dead-letter queue: a separate queue for messages a worker could not process after its retries are exhausted. It keeps a single poison message (a malformed payload, a template that always throws) from blocking the partition behind it forever, and it preserves the failures for inspection instead of losing them.

Deep dive 3: exactly-once is not available

The task is to prove exactly-once cannot be built here, choose which failure mode to accept, and make the surviving failure invisible to the user.

The proof. Sending is two steps: call the provider, then record locally that you called it. Those steps live in different failure domains (your database and Apple’s) so either can fail without the other, and nothing spans them: no transaction, no coordinator, no two-phase commit (the protocol where a coordinator asks every participant to prepare, then tells them all to commit). Nor can you repair the gap afterward by asking: no provider offers “did you deliver message X.” APNs has apns-collapse-id, but that collapses several notifications into one (it loses messages deliberately, the opposite of dedup) and FCM has no idempotency key at all.

So after an APNs send there are exactly three outcomes:

Unregistered is APNs saying the app was uninstalled; reaping the token means deleting it so nothing is ever sent there again. The third row is unresolvable by any amount of asking: the provider may already have delivered when the acknowledgement is lost.

sequenceDiagram
    participant W as Worker
    participant P as APNs / FCM
    participant D as Device
    W->>P: send(notification_id)
    P->>D: deliver
    P--xW: ack lost (timeout)
    Note over W: outcome unknown:<br/>delivered or not
    W->>P: retry (duplicate)
    P->>D: deliver again
    Note over D: client dedup on notification_id<br/>drops the repeat

Choosing which failure to accept. Since the two steps cannot be atomic, all you choose is their order:

• Commit then send gives at-most-once: die in between and the notification is marked sent although it never went. Silent, permanent loss, invisible to every dashboard.
• Send then commit gives at-least-once: die in between and it is re-sent on recovery. A duplicate: visible, annoying, and survivable.

Price both. Crash duplicates: every process death in the 50 ms window between the ack and the local commit re-sends what was in flight (750 sends at peak); at ~10 deaths/day across the fleet that is 7,500 duplicates/day. Ambiguous-timeout duplicates: 0.1% of 500 M calls end ambiguously and half of those were actually delivered (B9), so 250,000 duplicates/day. The overall rate is about one in 2,000, and 97% come from ambiguous timeouts, not crashes (250,000 / 257,500). That share is the useful part: tightening the crash window buys almost nothing, because the duplicates are inherent to the protocol, not to your code.

Making the duplicate invisible: at-least-once on the wire, dedup at the client. Every notification carries its notification_id; the device keeps a bounded set of ids it has displayed and drops repeats before rendering. At 1,000 ids × 8 bytes that is 8 KB per device, and this dedup set is exactly why the data model chose an 8-byte Snowflake id over a 16-byte UUID.

A server-side dedup store adds a second layer. It cannot collapse duplicates the provider created (those happened outside your system), but it does collapse the ones your own retries create. Its TTL comes from the retry ladder: the schedule of retry attempts, whose span here is 15 minutes. At 4x the ladder span (3,600 s), an hour of sends at 5,000/s is 18 M entries at ~66 bytes each = 1.2 GB in Redis, an in-memory key-value store with per-key expiry. A TTL shorter than the ladder lets the last retry through as a duplicate; a longer one is memory paid for with nothing to show.

The claim you are allowed to make is not exactly-once delivery. It is exactly-once as observed by the user, the property that actually matters. Two cases it does not cover, both acceptable but worth naming: a user with two devices sees it twice unless dedup is per user and per device, and reinstalling the app wipes the seen-set so the first re-send after a reinstall duplicates.

The mechanism appears in code below: a fake provider that delivers then withholds the ack, a deliver that sends before it commits, and a client inbox that drops repeats. The provider genuinely receives n-2 three times, yet the user sees three notifications, not five. The deliver function records "unknown", never "failed", because after a timeout you do not know which it was.

"""At-least-once on the wire, dedup at the client."""
import collections


class ProviderTimeout(Exception):
    """The call may or may not have delivered. Nothing can tell you which."""


class FakeProvider:
    """Delivers, then withholds the ack for ids listed in lose_ack."""

    def __init__(self, lose_ack=()):
        self.delivered, self.lose_ack = [], set(lose_ack)

    def send(self, notification_id, token, body):
        self.delivered.append((notification_id, token, body))
        if notification_id in self.lose_ack:
            raise ProviderTimeout(notification_id)
        return "accepted"


def deliver(provider, outbox, rec, max_attempts=3):
    """Send first, commit second. The other order loses messages silently."""
    for _ in range(max_attempts):
        try:
            provider.send(rec["id"], rec["token"], rec["body"])
        except ProviderTimeout:
            continue                      # ambiguous: retry, accept a duplicate
        outbox[rec["id"]] = "sent"
        return "sent"
    outbox[rec["id"]] = "unknown"         # never "failed" -- you do not know
    return "unknown"


class ClientInbox:
    """A bounded set of ids the device has already rendered."""

    def __init__(self, capacity=1000):
        self.seen, self.capacity, self.rendered = collections.OrderedDict(), capacity, []

    def on_receive(self, notification_id, body):
        if notification_id in self.seen:
            return False                  # duplicate: drop before rendering
        self.seen[notification_id] = None
        if len(self.seen) > self.capacity:
            self.seen.popitem(last=False)
        self.rendered.append((notification_id, body))
        return True


provider, outbox = FakeProvider(lose_ack={"n-2"}), {}
for nid in ("n-1", "n-2", "n-3"):
    deliver(provider, outbox, {"id": nid, "token": "t", "body": "hi"})

# The provider really delivered n-2 three times, once per ambiguous attempt.
assert [d[0] for d in provider.delivered] == ["n-1", "n-2", "n-2", "n-2", "n-3"]
assert outbox["n-2"] == "unknown"

inbox = ClientInbox()
accepted = [inbox.on_receive(nid, body) for nid, _, body in provider.delivered]
assert accepted == [True, True, False, False, True]
assert [r[0] for r in inbox.rendered] == ["n-1", "n-2", "n-3"]   # user sees 3

Retries

Toward APNs and Twilio you are the client, so the client-side retry rules from the rate-limiter chapter apply directly. Four inherited rules:

• Exponential backoff with full jitter, uniform(0, min(cap, base × 2^n)). Backoff means waiting longer after each failure; exponential means the wait doubles; full jitter means picking a random time between zero and that ceiling, not the ceiling itself. Backoff without jitter fixes nothing: a thundering herd of clients rejected in the same 10 ms window doubles their delays together and arrives together again.
• Honor Retry-After, but never sleep exactly it. FCM returns 429 with Retry-After; APNs signals overload with 429 TooManyProviderTokenUpdates and by resetting streams. Obeying the header literally builds the herd, because every blocked sender got the identical value.
• Retry at exactly one layer. Three tiers each retrying three times is 3 × 3 × 3 = 27 provider calls per notification during an incident.
• Keep a retry budget capped as a fraction of successes. A 10% budget turns a total outage into 1.1x normal load instead of 4x, because the budget is replenished by successes and during an outage there are none.

Two rules specific to notifications:

Not every failure is retryable, and classification matters more than the schedule.

Retrying a permanent failure six times is how a fleet spends its whole retry budget on devices that no longer exist.

The retry ladder must fit inside the TTL. A 15-minute ladder under a 60-second OTP TTL means every attempt after the first is dropped by the consumer before it is sent, which shows up as provider failure when it is really a config error. So ladders are per category: an OTP gets 3 attempts inside 60 s, order_shipped gets 6 attempts over 15 minutes.

Rate limiting, fatigue, and the opt-out path

Per-user rate limiting here is not an abuse control. It is a revenue control, and the arithmetic is the argument.

Notification fatigue is the effect: each extra message slightly raises the chance the user switches the channel off for good. If 40% of 100 M actives get 2 notifications past the third (B12), and each adds 5 basis points of opt-out probability (B6), that is 100M × 0.40 × 2 × 0.0005 = 40,000 opt-outs/day. At $5/year per live subscription (B5), a year of that destroys 40,000 × 365 × $5 = $73 M of run rate, 5.3x what the entire SMS optimization saves, and it appears on no infrastructure dashboard. An opt-out is permanent: you cannot win the user back with a notification, because that is the channel they just closed.

That $73 M is four soft factors multiplied, so treat it as an order of magnitude. The softest is the 0.0005, which must come from a holdout: an experiment where a randomly chosen slice of users is deliberately not sent the campaign, so the sent group’s opt-out rate has something to compare against. Without a holdout it is a guess, and the whole argument rests on it.

The per-category budget, enforced in the worker:

Quiet hours are a second, independent gate, and they create a load problem, because suppressing traffic overnight stacks it behind a deadline. One timezone at 30% of DAU with a 10-hour quiet window suppresses 30M × 5 × 10/24 ≈ 62.5 M notifications overnight. Release them all at the boundary and that is 62.5M / 60 s ≈ 1,041,667/s: a 69x spike over the system’s own peak, from one timezone alone.

The fix is the same uniform jitter used for retries: release at quiet_end + uniform(0, W), spreading each message over a window W. But W = 1 hour buys no margin. The design peak in exact seconds is 500M / 86,400 × 3 = 17,361/s, and a one-hour spread is 62.5M / 3,600 = 17,361/s, equal to the peak, to the digit. That is 100% utilization with a full night’s backlog still arriving: the same zero-headroom failure the push fleet has. W = 4 hours gives 4,340/s, a quarter of the peak, and leaves room for traffic that is not deferred. Ship the four-hour window. This also makes storing each user’s timezone a hard requirement: an unknown timezone gets the sender’s quiet hours, not none.

The opt-out check is the one control where a bug is priced in statutory damages: penalties fixed by law per violation. Under the TCPA (US Telephone Consumer Protection Act), damages for an SMS to a number that replied STOP start at $500 per message. A 0.01% leak against 25 M SMS/day is 2,500 messages, or $1.25 M/day of exposure, 6.4x the entire daily notification bill. Three consequences:

• The check runs in the worker, immediately before the provider call. A bulk fan-out computed 20 minutes ago has a stale view of consent, and 20 minutes is long enough for a user to have replied STOP.
• Inbound STOP, UNSUBSCRIBE, HELP are a write path into the preferences store, not a support ticket.
• The email unsubscribe link is one click: a plain GET, plus the List-Unsubscribe header mail clients turn into their own button. A flow that demands a login gets a spam complaint instead, and spam complaints get your sending domain blocked.

The code below implements the per-category budget and the quiet-hours release. Three guards matter because the obvious implementation is wrong off the default path: a quiet window that does not wrap midnight, a category the caller invented, and a u that is not a probability.

import collections


class Budget:
    """Per-user, per-category caps. Transactional is never capped, because
    the user's own action requested it; everything else is."""

    CAPS = {"transactional": None, "social": 3, "marketing": 2}
    WINDOW_S = {"transactional": 0, "social": 86_400, "marketing": 604_800}
    DEFAULT = "marketing"       # an unrecognised category is capped as marketing

    def __init__(self):
        self.log = collections.defaultdict(list)

    def allow(self, user, category, now):
        # An unknown category must have an ANSWER, not a KeyError. It falls to
        # the strictest cap and shares that bucket -- an invented name cannot
        # mint itself a fresh allowance.
        if category not in self.CAPS:
            category = self.DEFAULT
        cap = self.CAPS[category]
        if cap is None:
            return True
        recent = [t for t in self.log[(user, category)]
                  if t > now - self.WINDOW_S[category]]
        self.log[(user, category)] = recent
        if len(recent) >= cap:
            return False
        recent.append(now)
        return True


def release_at(hour_local, quiet_start=22, quiet_end=8, spread_h=1.0, u=0.5):
    """Quiet traffic is released across a spread window `spread_h` hours wide.
    Releasing it AT the boundary is a self-inflicted thundering herd."""
    if not 0.0 <= u <= 1.0:
        raise ValueError("u is a draw from uniform(0, 1)")
    if quiet_start < quiet_end:      # a window inside one day, e.g. 02:00-10:00
        quiet = quiet_start <= hour_local < quiet_end
    else:                            # a window that wraps midnight, e.g. 22-08
        quiet = hour_local >= quiet_start or hour_local < quiet_end
    return quiet_end + u * spread_h if quiet else hour_local


b = Budget()
assert all(b.allow("u1", "social", t) for t in (0, 10, 20))
assert not b.allow("u1", "social", 30)                 # 4th in a day: blocked
assert b.allow("u1", "social", 86_401)                 # window rolled
assert all(b.allow("u1", "transactional", t) for t in range(50))
assert [b.allow("u1", "promo_v2", t) for t in (0, 1, 2)] == [True, True, False]
assert release_at(14) == 14                            # daytime: send now
assert release_at(3, u=0.0) == 8 and release_at(3, u=1.0) == 9
assert release_at(3, u=1.0, spread_h=4.0) == 12        # the four-hour window
# A window that does NOT wrap midnight must not defer a midday message backward.
assert release_at(12, quiet_start=2, quiet_end=10) == 12
try:
    release_at(3, u=5.0)                                # u is a probability
except ValueError:
    pass
else:
    raise AssertionError("u outside uniform(0, 1) must be rejected")

Two of those guards are decisions, not padding. The midnight wrap: 22:00-08:00 wraps past midnight but 02:00-10:00 does not, and both are legitimate user settings. A single comparison chain that treats every window as wrapping would judge a noon message quiet under an 02:00-10:00 window and defer it to 10.5, nine and a half hours in the past. The Budget default: an unrecognised category must not raise, must not default to uncapped, and must not get its own private budget, or a caller could invent names to mint fresh allowances. Falling to the marketing bucket is the only option that fails safe, and this is the gate whose bug is priced in statutory damages.

Templates, localization, and the SMS encoding trap

A template is the message with holes in it: “Hi {name}, order {id} has shipped.” Rendering fills those holes for a specific user.

The whole corpus is small: 500 templates × 40 locales × ~2 KB = 40 MB. It lives in every sender process, so a template lookup is never an RPC (a network round trip dressed up as a function call). That closes the storage question in one line.

Render late, not early. Render at fan-out and the queue carries finished strings (~500 bytes each); render at send time and it carries a small reference: template id, user id, params (~120 bytes). Across 500 M/day that is 250 GB vs 60 GB in the queue, so 190 GB/day saved. The larger reason: a template fix applies to the backlog. The 6.9-million-message backlog from the queue section can still be corrected (a typo or broken link fixed mid-incident) if rendering is late. Render early and those 6.9 million strings are already written and unfixable.

Localization is not string interpolation. Three properties make it more than substitution:

• The locale belongs to the user, not the request. A locale is a language-and-region tag such as pt-BR. The order-shipped event knows nothing about what language its recipient reads.
• Plural and gender rules differ per language. The right tool is ICU MessageFormat (the International Components for Unicode message syntax, encoding rules like “one file / 2 files” per language), not %s plus an if that only handles English.
• The fallback chain is pt-BR → pt → en, never blank. And the template’s params_schema is validated at POST time, so a missing parameter is a 422 to the caller instead of a notification reading “Hi , your order”.

The SMS encoding trap. An SMS segment is the billing unit: a message longer than one segment is split and charged per part. (This is a different segment from an audience segment; always say which one.) GSM-7 is a 7-bit alphabet, so 160 of its characters fit in one segment. A single character outside it switches the whole message to UCS-2, the 16-bit Unicode encoding, which fits only 70 characters per segment, 67 once concatenation headers are added. So a 150-character message that was one segment becomes ceil(152 / 67) = 3 segments the instant one emoji is added, and the carrier bills per segment.

Price it: at a 10% share of the 25 M daily SMS carrying a non-GSM-7 character, each adding 2 extra segments, that is 2.5M × 2 × $0.0075 = $37,500/day, or $13.7 M/year, the same figure the entire fallback ladder saves. A curly apostrophe pasted from a word processor does this just as well as an emoji. So the segment count is a lint rule (an automated check that rejects the change) enforced in CI (the pipeline that runs on every commit) for every SMS template, with a live count in the editor.

The lint rule needs the full GSM-7 alphabet, or it is itself a bug. GSM-7 is not ASCII: the GSM 03.38 basic table includes £ ¥ § ¡ ¿, Greek capitals, and twenty-odd accented Latin letters (è é ù à ä ö ü ñ å Ä Ö Ü Ñ É ß Ç Ø), with € in the extension table. An ASCII-only approximation over-reports segments (it errs safe on money, not undercharging) but it rejects perfectly cheap German and French templates, which in a system localizing across 40 locales is the lint rule failing exactly the locales localization exists for.

GSM7 = set("@£$¥èéùìòÇ\nØø\rÅåΔ_ΦΓΛΩΠΨΣΘΞÆæßÉ"       # GSM 03.38 basic table
           " !\"#¤%&'()*+,-./0123456789:;<=>?"
           "¡ABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÑܧ"
           "¿abcdefghijklmnopqrstuvwxyzäöñüà")
GSM7_EXT = set("^{}\\[~]|€")                 # these cost two characters each


def segments(body):
    """One character outside GSM-7 drops the WHOLE message to UCS-2, from
    160 characters per segment to 70 (67 when concatenated)."""
    if all(c in GSM7 or c in GSM7_EXT for c in body):
        n = len(body) + sum(c in GSM7_EXT for c in body)
        return 1 if n <= 160 else -(-n // 153)
    n = len(body.encode("utf-16-le")) // 2   # UCS-2 code units, not codepoints
    return 1 if n <= 70 else -(-n // 67)


def daily_sms_cost(volume, emoji_share, price_per_msg=0.0075, body_len=150):
    plain = segments("a" * body_len)
    fancy = segments("a" * body_len + "\U0001f600")
    return volume * ((1 - emoji_share) * plain + emoji_share * fancy) * price_per_msg


assert segments("a" * 150) == 1
assert segments("a" * 150 + "\U0001f600") == 3      # one emoji, three segments
assert segments("cost: 5 {euros}") == 1             # {} are GSM-7 extended
# These ARE GSM-7; a carrier bills each as a single segment.
assert segments("£" + "a" * 159) == 1               # £ is in the basic table
assert segments("bestätigt " * 16) == 1             # so are ä ö ü é à ñ ...
assert segments("€" + "a" * 158) == 1               # € is extended: 2 chars
assert round(daily_sms_cost(25_000_000, 0.10)
             - daily_sms_cost(25_000_000, 0.0)) == 37_500

len(body.encode("utf-16-le")) // 2 counts UCS-2 code units, not Python characters, which is why one emoji costs two of the 70, not one.

Tracking, and why “delivered” is a lie

Three kinds of event get reported to every product team: sent, delivered, opened. Only one is real.

An SMSC (short message service centre) is the carrier’s store-and-forward node; a DLR meaning “the SMSC accepted it” says nothing about the handset. A tracking pixel is a 1×1 transparent image embedded in email; a download implies an open only if downloading the image implies a human looked.

Why the open rate is inflated. Apple Mail Privacy Protection (MPP), on by default in Apple Mail, pre-fetches every remote image regardless of whether the human opened anything, so every Apple Mail recipient registers as an open. If Apple is 55% of the base (reporting 100% opens) and the rest have a true 25% rate, the reported rate is 0.55 × 1.00 + 0.45 × 0.25 = 66.25%, inflated 2.65x. Worse, the inflation moves with Apple’s market share, so a year-over-year open-rate comparison is measuring device share, not campaign quality. Report the non-Apple cohort and its coverage, or do not report opens.

The metric that works: ask the device, not the provider. The client posts a receipt when it renders the notification, over the same channel that already carries the dedup key, so it costs nothing extra. That is 320 M receipts/day, one of the three events per notification inside a total event load of 15,000 writes/s, 3x the send rate. It gives a true rendered metric for the reporting population, and you must publish that population’s coverage next to the rate, because “80% of reporting devices rendered it” and “80% of notifications were delivered” are different claims and only the first is supported.

Dead tokens are the other silent corruption. Every uninstalled app leaves a token you keep pushing to. At a 1% monthly uninstall rate, the share dead after a year is 1 - 0.99^12 = 11.4%, so 11.4% of push volume (45.6 M/day) goes nowhere, and every rate computed against sent is 11% wrong. Reaping (deleting a token the provider declared dead) is not housekeeping: on 410 Unregistered (APNs) or NotRegistered (FCM), delete the token in the same code path that got the response. A second class of dead token, an app the user simply stopped opening, no provider will report, which is why the fallback ladder checks “seen in 30 days” instead of “token exists.”

Bottlenecks and scaling

Every resource the system consumes, with the number that bounds it, ordered by how much it should worry you.

Approximate counting means each node keeps its own per-user counter and syncs periodically, instead of coordinating on every send. The cap is then enforced within a small error, which is fine because the cap itself (“three social notifications a day”) is a product heuristic with a far wider error bar than the counter’s.

The scaling axis people reach for is sender throughput. The axis that matters is the queue partition count. A partition is one independently-consumed slice of a queue, drained in order, so a category’s whole latency budget is set by how much of the previous category sits in front of it. Adding senders does not fix that; adding partitions does.

Failure modes

Nine ways this design fails in production, laid out below. The Detection column is the one to study, because almost none of these is caught by an error rate, and a system whose only alarm is “errors went up” catches none of them.

To circuit-break is to stop calling a failing provider for a cooling-off period once its error rate crosses a line, so your own retries are not what keeps it down. A bounded pool per provider gives each provider a fixed allocation of slots, so exhausting Apple’s cannot consume the slots email is using.

Alternatives rejected

Each is rejected on a number or a named impossibility, not on taste.

• Synchronous send inside the caller’s request. Good: the caller learns the outcome immediately. Rejected: the outcome is not knowable, and an 833-second fan-out cannot live in an HTTP request. The caller gets 202 and a notification_id.
• One queue for everything. Good: one thing to operate. Rejected: a marketing backlog then sits in front of a 60 s OTP, since queue depth is shared. Partition by latency class.
• Exactly-once via a distributed transaction with the provider. Good: genuinely better. Rejected: it does not exist; no two-phase commit, no idempotency key, no query API. The alternative is imaginary, not merely worse.
• Server-side dedup only, no client dedup. Good: nothing to ship in the app. Rejected: the dominant duplicates (250,000/day vs 7,500) are created outside your system by ambiguous provider calls, and no server-side store can see them. The client is the only place that observes actual delivery.
• Retry until success on every failure. Good: maximum delivery. Rejected: Unregistered and PayloadTooLarge are permanent, so retrying spends the budget on garbage, and uncapped retries turn an outage into 4x self-amplified load.
• Render templates at fan-out time. Good: simpler worker. Rejected: it puts 250 GB/day of strings in the queue instead of 60 GB, and (the real reason) a template fix can no longer reach a 6.9 M-message backlog.
• Store tracking events in the outbox’s database. Good: one store, free joins. Rejected on shape and blast radius: 27.4 TB/year of append-only analytical rows against a small operational table, and one heavy query that saturates the store also stops the sending. Events go to a columnar store off a stream (see the database-internals chapter on why the layout differs).
• A single global per-user rate limiter with strict consistency. Good: exact caps. Rejected: the cap is a product heuristic with an error bar far wider than the limiter’s, so approximate counting with periodic sync is free and indistinguishable in outcome. Spend strictness on the opt-out check, which has statutory damages attached.

Final design

flowchart TD
    SVC["Internal services"] -->|"(user, category,<br/>template, params)"| API["API: validate,<br/>record idempotency key<br/>-> 202"]
    API --> FAN["Fan-out: resolve<br/>devices + preferences,<br/>render late"]
    FAN --> QT[["Transactional queue<br/>p99 < 30 s"]]
    FAN --> QM[["Marketing / social queues<br/>best effort"]]
    QT & QM -->|"durable, off-heap,<br/>per-category TTL"| W["Channel workers:<br/>bounded pool per provider"]
    W --> GATE{"Consent gate<br/>opt-out, quiet hours, budget<br/>-- last ms before send"}
    GATE -->|"suppressed"| REC["Record, do not send"]
    GATE -->|"allowed"| LADDER["Fallback ladder:<br/>push -> email -> SMS,<br/>escalate on evidence,<br/>by category"]
    LADDER --> PROV["APNs / FCM · SMS · SMTP<br/>at-least-once + dedup key"]
    PROV --> DEV["Device: client dedup<br/>+ rendered receipt"]
    DEV --> EVT[["Event stream<br/>columnar analytics<br/>15,000 writes/s"]]

    style GATE fill:#9d0208,color:#fff
    style QT fill:#2d6a4f,color:#fff
    style LADDER fill:#1d3557,color:#fff

Conclusion

A notification system is a fan-out engine whose last hop crosses into a third party you do not control, and that single property generates every hard problem here. The load-bearing takeaways:

• Exactly-once is not available. Choose send-then-commit for at-least-once, then make duplicates invisible with a client-side dedup key. About 1 in 2,000 messages duplicates, and 97% of those are inherent to the protocol.
• Cost drives routing. SMS is 5% of volume and 96% of the bill. A fallback ladder that escalates on evidence and by category, never on preference, saves ~$13.7 M/year at 20% deflection.
• Partition queues by latency class, not channel, so a slow provider or a marketing backlog never sits in front of an OTP, and so a slow APNs cannot stall email and SMS.
• The consent gate lives in the worker, in the last milliseconds before the send. A stale opt-out check is priced in TCPA statutory damages: $1.25 M/day at a 0.01% leak.
• Fatigue is the most expensive failure, ~$73 M/year and on no dashboard. Cap per category and measure with holdouts.
• “Delivered” is mostly a lie. Ask the device for a rendered receipt and always publish the reporting population’s coverage next to the rate. Reap dead tokens, which are 11.4% of push volume after a year.

Two small things carry outsized cost and belong in CI: rendering templates late (so a fix reaches the backlog), and an SMS segment-count lint using the full GSM-7 table (one emoji is $37,500/day).

One line to remember: the last hop always crosses into a third party you do not own, so you never promise exactly-once: you promise at-least-once, dedup at the client, route by cost, and check consent in the final millisecond before the send.

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications: at-least-once vs exactly-once, idempotence, and why distributed transactions across services are hard.
• Michael Nygard, Release It!: circuit breakers, bounded pools, and backpressure as stability patterns.
• Marc Brooker, “Exponential Backoff and Jitter” (AWS Architecture Blog): why full jitter beats plain backoff.
• Apple, Sending notification requests to APNs and Handling notification responses from APNs: the 200/4xx/Unregistered outcomes and token reaping.
• Firebase Cloud Messaging documentation: send behavior, error codes, and the absence of an idempotency key.
• Apple, Apple Mail Privacy Protection: why pixel-based open rates are unreliable.
• 3GPP TS 23.038 (GSM 03.38) and Twilio’s SMS character-encoding docs: the GSM-7 alphabet and how segments are counted and billed.

Related lessons: the rate-limiter chapter owns backoff, jitter, and the retry storm this design cites; the ID-generator chapter supplies the notification_id that doubles as the dedup key; the web-crawler chapter is the other design where politeness toward someone else’s server is the binding constraint.