How to design a chat system

Chat is a routing problem wearing a storage costume: messages are small and ordinary, but the destination is one socket on one machine that can die.

The one property

A persistent connection makes the serving tier stateful: the server keeps per-user state, so the box holding your socket (one open two-way connection) is not interchangeable.
Every hard thing descends from this: locating a user’s socket, ordering across writers, cheap presence, and a delivery guarantee that cannot be exactly-once.
Presence = the online/offline/last-seen dot. Fanout = getting one message out to many recipients.

Decision	Choice	Why
Transport	WebSocket	Persistent push channel; makes tier stateful
Store once or per-recipient	Once per conversation	Per-recipient = 7.9x bytes, 86.5 PB
What orders messages	Per-conversation `seq` counter	Wall-clock skew inverts order

Scale: 500 M DAU, 4% concurrent = 20 M sockets; 40 msgs/user/day = 20 B/day = 231 k msgs/s avg.
Fanout: 0.7 x 1 + 0.3 x 24 = 7.9 recipients/msg -> 1.83 M deliveries/s (the number that sizes registry, receipts, fanout).
Storage: 100 B/row, 2 TB/day, x3 replicas x5 yr = 11 PB. Chat is not storage-bound.

Four resources each floor the box count; the largest wins (all must hold at once):

Resource	Floor	Note
Memory	~3 boxes	10 KB/socket tuned; footprint spans ~14x, a config decision
File descriptors	~20 boxes	`fs.nr_open` ~1.05 M per process
Proxy port exhaustion	306 tuples	65,535 source ports per (proxy IP, backend IP, port)
Blast radius	100 boxes	200 k sockets/box; a death orphans 200 k clients

Death is bounded by the session registry, not TLS handshake (~1.5 ms, cheap).
Reconnect jitter window = blast radius / op budget = 200 k / 10 k ops/s = 20 s. Jitter = each client waits a random delay before retrying.

Server timestamps fail: clock-diff SD = 10 x sqrt(2) = 14.14 ms; a 50 ms gap is 3.5 SD -> ~1 in 4,900 inversions -> ~203,000 inverted pairs/day. Snowflake ids just re-encode the same skew.
Fix: one monotonic counter per conversation, assigned by the single owner shard (seq = last_seq + 1).
Not a bottleneck: mean conversation gets ~1 msg / 8 hours (0.000037/s); contention bounded by one conversation, never the platform.

`seq` buys	vs
Total order	Timestamps + tiebreak (disagree across gateways)
Gap detection (holds 41+43, knows 42 missing)	Hope / full resync
Idempotency: `(conv_id, seq)` unique	Nothing client-side
O(1) cursors: unread = `last_seq - last_read_seq`	A set of msg-ids per user

Storage fanout: never per-recipient. Socket fanout: always, but strategy flips.
Direct routing costs M sends; broadcast costs G gateways, where G = N x (1 - (1 - 1/N)^M), capped at N = 100.
Threshold where (1 - 1/100)^M = 0.01: M = ln(0.01)/ln(0.99) = 458. Below = direct; above = broadcast. Set by fleet size (halve fleet -> 228).
Above threshold also: suppress receipts, suppress presence, membership becomes the topic (a “group” and a “channel” are different systems).

Exactly-once is unachievable (two-generals problem): a lost ack is indistinguishable from a lost message, so the sender must retry.
At-least-once retries until acked (never lost, may duplicate); receiver dedups on (conv_id, seq). “Exactly-once” = this, honestly named.
Volume: ~0.3% ack-loss x 1.58e11 deliveries = ~474 M duplicates/day. Dedup is a mainline path.
Two dedup points: sender keyed on client_msg_id (a client-generated ULID/UUIDv7 idempotency key, ~1.67 GB Redis, 300 s window); receiver keeps last 1,000 seq/conversation (~800 KB on a phone).
Receipts are cursors, not events: one read per open carrying highest seq -> 3.66 M/s down to 87 k/s (42x).
Offline queue is a query, not a store: range scan from last_delivered_seq (a real store would be 1.05 TB); offline users just get a content-free push wake-up.

Heartbeat cost = connections / interval = 20 M / h. Detect offline after 2 missed beats; mean staleness = 1.5h.
h bounded by battery (radio tail ~10 s -> duty cycle 10/h) below and carrier NAT timeout (~300 s, else silent half-open) above. Window ~120-240 s; 180 s in the middle.
The green dot comes from activity (any frame refreshes presence), not heartbeats.
Fanout fix subscribe-on-view: send transitions only to the ~1.5 watchers with the conversation open -> 1.16 M/s down to 87 k/s (13x); token-bucket per user to tame flapping.

The two routing diamonds ask unrelated questions: “how to route” (member_count vs 458) vs “is this recipient reachable now” (socket open?). Conflating them is the classic drawing error.
Session registry locates sockets (where a user is); consistent hashing owns conversations (where they should be). Different questions.
The message store is the only authoritative copy: every failure degrades to “late”, not “lost” — one log + one cursor per client turns every failure into a reconnect.