Scaling a service from one user to millions

Never add a box until a number crosses a line, and always be able to name the number: size the workload into a request rate, the rate into cores, cores into machines, then climb the ladder one rung at a time.

The sizing loop

The method never changes: turn a user count into a rate, a rate into cores, cores into machines.

DAU x actions / 86,400 s  ->  avg QPS  x3 peak  ->  cores = QPS x S / 0.7  ->  machines

• DAU daily active users; QPS queries (all requests) per second.
• Peak = 3x average here (an assumption): push-notified feeds fan out and burst; smooth traffic is ~2x.
• Every rate carries two labels or it lies: peak vs average (peak sizes anything with a queue in front — cores, replicas, connections; average sizes what accumulates — bytes, egress bills), and offered load vs service capacity (never divide two offered loads and call it utilization).
• Sample workload: 20 req/DAU/day, 10% writes (9:1 read:write), 8 ms app CPU + 5 ms DB CPU per request.

The one law that forces the ladder

Model each tier as an M/M/1 queue. Utilization rho = lambda x S / c (arrival rate x service time / servers) is a cliff, not a slope.

mean = S / (1 - rho)        p99 = 4.6 x S / (1 - rho)      (4.6 = ln 100)

• Target rho <= 0.7: a 21% surprise merely doubles p99. “20% CPU headroom” at 90% is not headroom.
• S and (1-rho) each appear once: faster code and less load pull the same lever.
• A 17 ms service time at 80% util is a 391 ms p99 — high p99 + high CPU means utilization is the problem, nothing to profile.

The ten-rung ladder

Climb in order, never skip, each forced by one measurement.

Formulas to memorize

• Cache hit rate (Zipf-1): ln(k) / ln(N), where N counts cache keys, not rows. Caching 1% of 10M timelines = 71% of reads; 99% hit needs k = 8.5M (85% of the keyspace).
• Little’s law: L = lambda x W (in-flight = rate x duration). Sizes threads, queue backlog, and revocation-list size.
• Buffer hit rate matters 1,000x: page in memory 0.1 us vs disk 100 us; a 9-point hit-rate drop is ~9x latency.
• Machine count: cores = peak_QPS x S / 0.7; add availability tax / (2/3) to survive losing 1 of 3 AZs (20 -> 30).
• Availability: model gives 7 nines from 2 boxes; reality gives ~4 (99.99%) because failures correlate (bad deploy, AZ outage, cert expiry).

Gotchas

• N+1 query: an ORM lazy-loading 300 followees one row at a time = 150 ms of pure round trip; cheap in bulk, ruinous one row at a time.
• Replica lag is a lost-update bug, not staleness: a read off a lagging replica feeds a write to the primary. Fix with an LSN token (write returns log position; read falls back to primary until a replica has replayed past it), or sticky-primary window (11% of read traffic, not 100%).
• Cache write path: db.insert before cache.delete, and delete (never set) — DB is the only source of truth.
• Thundering herd: hot-key expiry needs single-flight; synchronized mass expiry needs TTL jitter — two failures, two guards.
• Size the cache for the miss path: a cold cache instantly demands 6 replicas, not 2; a flush is an outage unless warmed.
• Shard last (irreversible): route on hash(id) mod 1024 + lookup table (adding a node moves ~6%), never mod 16 (~94%). Shard by user_id, not post_id, to avoid scatter-gather. The migration (dual-write -> backfill -> checksum -> flip) is the project.
• Logging bill is fan-out, not line size: 1 KB/req = $600/mo, but 20 KB across 20 services = $12,000; head-based sampling (keep all errors + p99 tail, 1% of the rest) = 2.1% of the bill, loses nothing.