How to design a news feed — cheat sheet — System Design

A feed turns on one question, materialize a timeline at write or at read, and the answer is always a hybrid because push dies on an unbounded tail and pull dies on latency.

The one decision

Fan-out on write (push): on publish, write a pointer into every follower’s inbox. Read = one lookup.
Fan-out on read (pull): publish writes only the author’s outbox. Read fetches + merges every followee’s outbox.
Push pays once per follower at write time (off critical path, async behind a queue). Pull pays once per followee at read time (all on critical path).
Answer = hybrid: push for the 99.99% of authors, pull/broadcast for head accounts.

push cost/day = F x p     pull cost/day = F x r
push cheaper  <=>  p < r        (F cancels)

Numbers that drive it

Quantity	Value
DAU / posts per day	2e9 / 5e8
reads per user per day (r) / posts per day (p)	5 / 0.2
median follows (pull cost) / mean followers (push cost)	200 / 400
push load	2.0e11/day, 2.31 M/s avg, 5.79 M/s peak
pull load	2.0e12/day, 23.1 M/s (10x push, all critical)
post-hybrid push	9.0e10/day (55% removed), 1.04 M writes/s
inbox entry	24 B (post_id 8 + author_id 8 + created_at 4 + flags 4)

Break-even vs real threshold

Break-even is p < r, follower count cancels: push infrequent posters, pull prolific ones. At r=5: median author (0.2/day) push 25x cheaper; news wire (40/day) pull 8x cheaper.
Implemented threshold is on followers anyway: pull is cheap only when the pulled set lives in-process RAM. RPC ~500 µs vs in-process read ~100 ns (5,000x).
RAM budget ranks accounts by followers. Threshold 1 M followers = 50,000 accounts = 80 MB hot set, fits in process. RAM scales T^-2.5, so halving T multiplies memory 5.7x; one decade below 1 M = 25 GB, no longer fits.

Why each pure design fails

Symptom	Root cause
Celebrity fanout: 1e8 writes/post	Push cost scales with power-law follower count
Read fan-in: p99 400 ms vs backend 20 ms	Latency is max over 200 fetches; `0.99^200 = 0.134`, so 87% of reads hit a straggler
Pagination dup	`OFFSET` counts rows in a list being prepended to
Cache loss: 10x DB jump	Load behind cache is `(1-h)`; losing it multiplies by `1/(1-h)`

Celebrity burst: one 100 M post is 250,000x mean write budget. Delivery time = burst / spare capacity (not / load). At peak+25% headroom: 20 s avg, 69 s peak; unbounded if fleet sized at peak demand. Hybrid removes the event, never schedules it.

Inbox store

Derived, disposable pointer index. Post store is authoritative; losing an inbox entry costs one missing post, rebuilt from post store + graph.
Cap 500 entries: storage arg is weak (36 TB, easy); bandwidth binds (whole slice ships to ranker every read, linear in cap); trim lazily at cap+100 (1% overhead). TTL 72 h binds first for median users; cap binds for high-connectivity tail.
Shard count: capacity binds (36 TB / 1 TB = 36 shards), not IOPS (~3 shards on NVMe at 500k IOPS). x3 replicas = 108 boxes.
LSM tree turns 25 MB/s random writes into 250 MB/s sequential; ~10x write amplification, still beats a B-tree that rewrites a whole page per 24 B append.

Pagination

Cursor, never offset. OFFSET after k head inserts: last k of page 1 served again (k dups); k deletes = k skipped.
Corruption chance = 1 - exp(-lambda t), t cumulative. Median 3.5% page 2; p95 user 14.8% page 2 → 55% page 5. ~100 M wrong pages/day platform-wide.
Keyset cursor for chronological: WHERE (created_at, post_id) < (?, ?), composite index on (user_id, created_at, post_id).
Ranked feeds: scores move (model version, decay, viewer state), so a post crosses the boundary twice. Fix = freeze one snapshot per session, cursor = integer position. 27.8 M concurrent sessions (Little’s law) x 800 B = 22 GB Redis, TTL 300 s.
Validate cursor: cursor is None not if not cursor (0 is a real position); cap limit server-side (50).

Cache tiers and hit-rate economics

Tier	Contents	Size	h
L0	Head hot set (50k) in process	80 MB	never misses (broadcast)
L1	First 2 pages / DAU inbox, RAM	1.92 TB	—
L2	Hydrated post objects	225 GB	0.891
L3	Media on CDN	—	—

Hydrate = replace 8 B pointer with 1.5 KB object.
Zipf exp 1: h = ln(k)/ln(N). Load behind cache = 1/(1-h): h=0.782 → 4.6x, h=0.891 → 9.2x, h=0.99 → 100x. Hit rate rises with log of memory but load falls as reciprocal of remainder, so returns don’t diminish. Stops when cache ≈ full corpus (no longer a cache).
Failure runs backward: losing L2 is a 1/(1-h) step function (9.2x instant), not a ramp. Fix: consistent hashing, request coalescing (single-flight), staggered restarts.

Gotchas / degradation

Every failure gives up a property, not the service: dead ranker → reverse-chronological; lost snapshot → keyset cursor, scroll resets to top; dead inbox shard → post store + graph; stale hot set → pull RPC. Ranking is a quality dependency, not availability.
Egress 69.4 Gbps peak; gzip → 23.1 Gbps; media is a URL, never on this path.
Graph stored twice (denormalized): followee→followers for fanout, follower→followees for pull; failure mode is one-sided edge, fixed by reconciliation job.
Author needs read-your-writes on own post; everyone else eventual.