Scaling a service from one user to millions

In this lesson, we’ll scale one ordinary web app from a single machine to ten million daily users. There are ten architectural moves between those two points, and we’ll make each one only when a specific measurement crosses a line, never because a design looks elegant. By the end you’ll be able to size each tier from a workload, name the number that forces the next move, and defend the whole ladder in an interview.

Along the way we’ll pick up the standard vocabulary of scaling: load balancer, read replica, cache, content delivery network, stateless tier, message queue, shard. We define each term the first time we need it.

The method never changes. We take a traffic figure, turn it into a number of machines, and name the measurement that forces the next move. What we end up with is an architecture with a number under every box: thirty web machines, one primary database, two read replicas, a cache holding 0.8 GB, a queue and its workers, and for each of them, the measurement that put it there.

The whole argument runs on eight workload numbers: how many people use the product each day, how many requests each makes, how much processor time one request costs in the application and in the database, and how many bytes it stores.

Six words carry every figure below

We quote every figure below in these six terms, so it pays to fix them now.

Two of these deserve more than a line, because interviews trip on both:

• Latency and throughput are different quantities, and a system can be excellent at one and terrible at the other. Picture a system that handles 10,000 requests per second where each takes two seconds: great throughput, terrible latency. Keeping them apart is what stops you sizing a queue with a latency number.

• p99 summarizes a set of measurements. Sort the day’s requests slowest-last and read the value one percent from the slow end. A p99 of 5.1 ms means 99 requests in 100 finished within 5.1 ms and one did not. We reach for percentiles instead of averages because one request in a hundred being slow is exactly what a user notices, and an average buries it.

The arithmetic style behind every estimate here is drilled in Back-of-the-envelope estimation. The database mechanisms we lean on, B-trees, replication modes, isolation levels, shard keys, are derived in Database internals. Neither is required reading: wherever we need one of their results, we restate it in a sentence first and link second.

The workload, stated once

Before we size anything, we pin down the inputs. Eight numbers drive everything we compute, and we fix them here so no later step can quietly introduce a ninth. We quote them per DAU, because daily active users is the one figure a product manager can actually hand you. Treat this block as the full brief: nothing else about the product is known.

APP            a social feed: post, follow, read a home timeline
REQUESTS       20 per DAU per day, of which 10% are writes
PEAK           3x average          (an assumption, defended below)
APP CPU        8 ms per request
DB CPU         5 ms per request, averaged over the mix
POST SIZE      400 B stored
FOLLOWS        300 per user, mean
COMPUTE        $0.05 per vCPU-hour

Why hardware constants live on their own list

The eight numbers above describe the users. The constants below describe the machines. We keep the two lists apart on purpose: a workload assumption is something an interviewer can push on, a hardware constant is not, so it matters which one a calculation just leaned on. Five constants carry most of the lesson.

About ten more, page-read latencies, network-card throughput, instance ceilings, per-gigabyte prices, appear once each, right where we use them.

Why the peak multiplier is 3x

The PEAK line says the busiest second of the day carries three times the traffic of the average second. It matters because we buy machines for the busy second, not the average one, so this one number multiplies every capacity figure we compute. It is also the only one of the eight that is an assumption rather than a measurement, so we state it out loud.

Here’s the intuition. Smooth consumer traffic with one broad rise and fall across the waking day peaks near 2x. Bursty or event-driven traffic runs closer to 3x. Our feed is push-notified, and a notification fan-out, one event firing a message to many users at once, concentrates arrivals into minutes instead of spreading them across an evening, so we take 3x. A different product would justify a different multiplier and a correspondingly different fleet size. The number is defensible; quoting it silently is not.

The read:write ratio comes from assumed behavior

The REQUESTS line says 20 requests per user per day with 10% writes. That is 2 writes per user per day and a 9:1 read:write ratio, and it quietly drives every rung below: the 694 peak writes/s at 10M DAU, the 8 GB/day hot set, and the replica-apply wall at ~30M DAU all divide by those 2 writes/day. Get the ratio wrong and every downstream number moves with it.

A different assumed posting rate gives a different ratio and a different architecture. A read-heavy feed like this one caches computed timelines. A design where each post reaches far more readers pushes you toward fanning out on write instead, which is the subject of How to design a news feed. So we state the assumed user behavior before we quote a ratio: a high-write feed and a low-write one are different products wearing the same diagram.

Where the 5 ms of database time goes

The 5 ms of database processor time is an average over cheap and expensive queries, and averages hide the thing we need. Splitting it into its parts is what lets us size replicas and caches later, because those tiers each target one part. We multiply each kind of request by how often it happens, and the products sum to 5 ms.

per request, averaged over the request mix
  3 point lookups x 0.3 ms                        =  0.9 ms
  0.2 timeline queries x 18 ms                    =  3.6 ms
  0.1 writes x 5 ms (row + 3 indexes + WAL)       =  0.5 ms
                                                     -------
                                                     5.0 ms
  of which reads                                  =  4.5 ms   -> 90%

Two terms come from the row + 3 indexes + WAL note on the write line, and both matter later.

An index is a side structure the database maintains so it can find rows by a column’s value without scanning the whole table. Keeping three of them current is why one logical write costs four physical ones.

The write-ahead log (WAL) is the fifth write. Before a row is changed in place, the change is appended to a log file and forced to disk, so a crash halfway through can be replayed forward from the log. That same log is the stream replicas consume, so every replication and read-your-writes mechanism later in this lesson operates on it. Hold onto that; it is the thread running through steps 3 and 7.

Where the 8 ms of application time goes

The application’s 8 ms splits the same way, and this is the number that will size the entire web fleet, so it earns the same breakdown.

per request, averaged over the same mix
  parse, authenticate, route                      =  1.0 ms
  marshal 3 queries + deserialize 3 result sets   =  1.5 ms
  0.2 timeline assemblies x 20 ms of templating   =  4.0 ms
  serialize + gzip the response                   =  1.5 ms
                                                     -------
                                                     8.0 ms

Half the app tier is one endpoint, the timeline, the same shape as the database’s 3.6-of-5.0 ms. Notice the pattern: every layer of this lesson ends up paying for that one endpoint.

Where the 18 ms timeline query comes from

We derive the 18 ms rather than assume it, because it is the only expensive operation in the system and every later rung is trying to avoid running it. If you know what it is made of, you know what each tier is buying you out of.

Building one home timeline means finding the newest posts written by each of the 300 people a user follows and merging them into one list. The database finds each of those 300 starting positions by walking down a B-tree index, a shallow tree whose leaves point at rows, so locating one row costs about four page reads (a page is the fixed 8 KB block a database reads as a unit) instead of scanning the whole table.

merge the 20 newest posts from each of 300 followees
  300 index descents x 4 pages each               =  1,200 page reads
  at 97% buffer-cache hit, 0.1 us hit / 100 us miss
    1,200 x (0.97 x 0.1 + 0.03 x 100) us          =  3.7 ms of IO wait
  heap fetches, top-k merge, serialize                14.3 ms of CPU
                                                      -------
                                                      18.0 ms

Two constants there come back to bite or save us later. Four pages per B-tree descent holds because a tree with a few hundred children per node reaches billions of rows in four levels (Database internals). And 0.1 microseconds against 100 microseconds is the gap between finding a page in the database’s buffer cache, the slice of memory holding recently used pages, and fetching it from disk. That 1,000x ratio reappears at every layer, because every layer is making the same bet: keep the working set one tier up. That single idea is what the rest of the ladder keeps cashing in.

The traffic ladder turns users into a request rate

This table is the workhorse: it turns each user count into a request rate, and every later step reads its numbers straight off it. Each row is the same two operations. We divide the day’s requests by 86,400 seconds to get the average, then multiply by 3 for the peak. Let’s do it once by hand at 10M DAU so the table is not a black box:

requests/day  =  10,000,000 DAU x 20 requests  =  200,000,000 /day
avg QPS       =  200,000,000 / 86,400 s        =      2,315 /s
peak QPS      =  2,315 x 3                     =      6,944 /s
reads/s peak  =  6,944 x 0.90                  =      6,250 /s
writes/s peak =  6,944 x 0.10                  =        694 /s

Every rate carries two labels, or it lies to you

A rate stated without these two labels is the single most common way a capacity calculation ends up wrong by a factor of three. Attach both, every time.

Peak or average. Peak sizes anything with a queue in front of it, processor cores, replicas, connections, workers, because those must survive the busiest second. Average sizes anything that accumulates, stored bytes, egress bills, because those only care about the monthly total. Every rate below is peak unless it says avg. Compare one of each and you are off by exactly the 3x factor that separates them.

Offered load or service capacity. Offered load is what the workload asks for (6,944 peak req/s). Service capacity is what a machine supplies (4 cores x 0.7 target / 0.008 s per request = 350 req/s, where a vCPU is one virtual core as a cloud provider sells it). Which one you divided by tells you what question you actually answered:

Never divide two offered loads and call the result a utilization. A fleet whose capacity exactly equals peak offered load runs at 100% utilization, written rho = 1, and running there costs far more than the last 30% of headroom suggests. That cost is the whole next section.

The one law that forces most of the ladder

One equation decides how busy a machine can safely get, and it is the reason seven of the ten rungs exist. The key claim is this: utilization, the fraction of the time a machine is busy rather than idle, behaves as a cliff, not a slope. Every “we’re at 80% CPU, we’re fine” incident traces back to missing that. Let’s build the intuition, then the formula.

The model: a single-server queue

We model a tier as an M/M/1 queue. That name is Kendall notation, a shorthand for the model’s assumptions:

• The first M: arrivals are Poisson, meaning requests arrive independently, and how long you have waited tells you nothing about when the next comes.
• The second M: service times are drawn from the same kind of distribution.
• The 1: one server.

Two symbols carry the arithmetic: S is the mean service time (how long one request occupies the server once it starts, no waiting counted), and lambda is the arrival rate. Utilization is just a rate times a duration:

rho  =  lambda x S / c        (c servers; c = 1 for M/M/1)

rho has no units. Read it as the fraction of time a server is busy. For the 8-vCPU database primary at 1M DAU (694 peak req/s, 5 ms each):

rho  =  694 /s  x  0.005 s  /  8 cores  =  0.434

From utilization to p99

Here is where utilization turns into something the user feels. For M/M/1 the response time, queue wait plus service, is exponentially distributed with mean S/(1-rho). An exponential with mean m has its q-th quantile at m x ln(1/(1-q)), and ln(1/(1-0.99)) = ln(100) = 4.6. So the two formulas the rest of the lesson leans on are:

mean   =  S / (1 - rho)
p99    =  4.6 x S / (1 - rho)

Worked on the same primary. S is how long a request occupies the tier, and the tier has 8 cores in parallel:

requests the box finishes flat out  =  8 cores / 0.005 s  =  1,600 /s
S = 1 / 1,600 s                                           =  0.625 ms

1 - rho  =  1 - 0.434   =  0.566
mean     =  0.625 / 0.566   =  1.10 ms
p99      =  4.6 x 1.10       =  5.1 ms

Read the formula as advice: S appears once and (1-rho) appears once, so making the code faster and taking load off the tier pull the same lever, and only one of them requires a deploy. That is worth remembering when the pager goes off.

What the cliff actually looks like

The last column is the one that should change how you behave. Doubling the p99 means doubling the queue multiplier, which needs rho' = (1 + rho)/2; the traffic increase to get there is rho'/rho - 1. At 50% utilization it takes a 50% traffic increase to double the p99; at 90% it takes 5.6%. That is why every capacity number in this lesson targets rho <= 0.7, and why “we still have 20% CPU headroom” is not a true statement about headroom at all.

The formula as code you can run

These two functions turn the formulas into something runnable: the first gives a p99 from a service time and a utilization, the second gives a core count from a request rate. Both lean optimistic on two counts, and it is worth knowing which way they lie: modelling c cores as one c-times-faster server queues less than c real servers would (the M/M/c model), and real arrivals are burstier than Poisson. Both errors point the same way, so a measured p99 comes out worse than these formulas, never better. Use them to find the cliff, not to promise an SLO (service level objective, a published target you are held to and must therefore measure).

def p99_ms(service_ms: float, utilization: float) -> float:
    """M/M/1 p99 response time. The 4.6 is ln(100)."""
    if not 0 <= utilization < 1:
        raise ValueError("utilization must be in [0, 1)")
    return 4.6 * service_ms / (1 - utilization)


def cores_needed(peak_qps: float, service_ms: float, target_util: float = 0.7) -> float:
    """Cores to hold `peak_qps` at `target_util`. 0.7 is the default because
    that is where a 21% traffic surprise merely doubles p99."""
    return peak_qps * (service_ms / 1000.0) / target_util

Three of the chapter’s headline numbers fall out of them:

p99_ms(0.625, 0.434)          # 5.1 ms  -- the primary, comfortable
p99_ms(0.625, 0.694)          # 9.4 ms  -- p99 nearly doubled for a 26-point CPU move
cores_needed(6944, 8) / 4     # 19.8 -> 20 four-vCPU web instances at 10M DAU

One insight to carry into interviews: a 17 ms service time at 80% utilization is a 391 ms p99 (p99_ms(17, 0.8)). So when a p99 is 400 ms and CPU is 80%, the utilization is the problem, not a slow function, and there is nothing to profile. That single sentence resolves a whole class of “why is it slow” incidents, and it is what forces the ladder we are about to climb.

The ladder

Here are the ten rungs in the order traffic forces them. We climb them as the product grows, and we never skip one. Each rung is labelled with the measurement that pushes us onto it.

flowchart TD
    S0["0 · Single box<br/>app + DB + files<br/>up to ~100k DAU"]
    S1["1 · Split the DB<br/>forced by a RAM ceiling"]
    S2["2 · Load balancer + N web<br/>forced by availability"]
    S3["3 · Read replicas<br/>forced at 1.6M DAU"]
    S4["4 · Cache<br/>forced by 7 copies of one hot set"]
    S5["5 · CDN<br/>forced by 28 Gbps of static bytes"]
    S6["6 · Stateless tier + session store<br/>forced by the error budget"]
    S7["7 · Multi-region<br/>Europe pays 450 ms"]
    S8["8 · Message queue<br/>forced by 70 threads for 1% of traffic"]
    S9["9 · Shard<br/>forced at ~30M DAU by replica apply"]

    S0 --> S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 --> S9

    style S0 fill:#1d3557,color:#fff
    style S9 fill:#9d0208,color:#fff

Every rung in one sentence

Three terms the diagram leans on first

Availability is the fraction of the time the service is actually answering. We quote it as a run of nines: 99.9% is about 8.8 hours a year of not answering, 99.99% about 52 minutes. It is not the same quantity as throughput and you do not buy it the same way; step 1 prices a single box at 99.965% and shows that 57% of that loss is deploys, which no amount of extra capacity removes. Availability is the forcing function for rung 2 and half of rung 7.

A RAM ceiling, not a CPU one (rung 1): on one box, the application’s heap, the memory it allocates for its working data, and the database’s page cache compete for the same 16 GB. The loser is whichever you notice second.

Europe pays 450 ms (rung 7): three serial round trips to fetch the page plus three more for the API calls the page needs (an API is one service’s programmatic interface, the thing other code calls), at 75 ms each. No amount of extra capacity in one region removes those milliseconds; only a second region does.

Step 0 — One box, and how far it actually goes

We start where every product starts: one box. A single ordinary machine carries far more traffic than most estimates assume, and knowing the real number is what stops you over-designing on day one. The setup is one machine with 4 vCPU and 16 GB of memory running the application, the database and the static files together. Let’s compute how far it gets us.

How much one box carries

app tier   =  4 x 0.7 / 0.008  =  350 req/s at rho 0.7
db tier    =  4 x 0.7 / 0.005  =  560 req/s at rho 0.7
combined, 8 ms app + 5 ms db = 13 ms of CPU per request
           =  4 x 0.7 / 0.013  =  215 req/s

The combined line is the honest one: both tenants share the same 4 cores and their costs add, so we size on 13 ms, not 8. Now walk 215 peak req/s back to a user count by running the traffic ladder in reverse:

215 / 3 = 72 avg req/s  ->  72 x 86,400 = 6.2M requests/day  ->  / 20 = 311,000 DAU

One box carries about 311,000 DAU. At 100k DAU the ladder offers it 69 peak req/s against 215 capacity, which is 22% utilization, so the correct architecture there is one machine and a backup script. Resist the urge to draw anything fancier. But something does eventually break on this box, and it is not the CPU.

Memory breaks first, not CPU

The two tenants want opposite things from the same 16 GB. The application wants heap; the database wants page cache, the memory in which it keeps recently used disk pages so it does not read them again. The database’s appetite is 25–80% of memory depending on the engine (Database internals).

What has to fit in page cache is the working set, the portion of the data actually being read. Let’s trace how long it stays inside 16 GB at 100k DAU:

posts        100k DAU x 2 posts/day x 400 B      =    80 MB/day
follows      100k users x 300 x 16 B x 3 (index) =   1.44 GB
users        100k x 1 KB                         =   0.1 GB

available page cache = 16 GB - 4 (app heap) - 2 (OS)  =  10 GB
  follows + users already occupy                   =   1.54 GB
  left for posts                                   =   8.46 GB
posts cross it after 8.46 / 0.08                   =   106 days

One follow edge is two 8-byte identifiers (16 B), and the x 3 is the row plus the two index entries that answer “who does X follow” and “who follows X”. Follows and users are resident from day one, so we subtract them before dividing; dividing the whole 10 GB by 80 MB/day instead gives 125 days, which is 18% optimistic about the only deadline that matters here.

Why does crossing that line hurt so much? Because it is not gradual: a page in memory costs 0.1 microseconds and a page from disk costs 100. Watch the same 1,200-page timeline query at four buffer hit rates (the fraction of page requests served from memory):

A 9-point drop in hit rate is a 9x latency increase. Nothing about the query changed; the working set simply left RAM. That 106-day countdown is what forces the next rung.

Step 1 — Split the database off

When the 106-day deadline arrives, the first machine we add is a second one for the database, not for the application, taking us from one machine to two. Nothing runs faster on the day we make the change; what we buy is four things:

The cost: every query now crosses a network

Nothing is free. On one box a query was a function call. Now it is a message and an answer, 0.5 ms, and the timeline path issues 3 queries:

added latency  =  3 x 0.5 ms  =  1.5 ms per request  =  1.5 / 18  =  8.3% of the timeline query

Paying 8.3% for those four benefits is a good trade. The version that is not fine involves an ORM (object-relational mapper, the library that lets application code treat rows as objects). Left alone, an ORM will lazy-load, fetching each related row the moment the code touches it, so asking for 300 followees one at a time becomes 300 messages, 300 x 0.5 ms = 150 ms of pure round trip. That pattern is the N+1 query: one query for the list, then one more per member. It was invisible on one box and now it is the entire latency budget. The lesson to keep: the network is cheap in bulk and ruinous one row at a time.

Vertical or horizontal: reach for the bigger box first

Vertical scaling means a bigger machine; horizontal scaling means more machines of the same size. Do not reach for horizontal by reflex. Vertical is one line of config and its range is surprisingly large:

one 4-vCPU box   =  4 x 0.7 / 0.008    =    350 req/s
one 192-vCPU box =  192 x 0.7 / 0.008  = 16,800 req/s   ->  48x

A failure domain is a set of things that go down together; a single machine is one. A failover is promoting a standby when the live machine dies. So the binding ceiling here is availability, not throughput. Let’s price the single box directly and see it:

planned:   2 deploys/week x 60 s        =  104 x 60      =   6,240 s/yr
unplanned: 4 incidents/yr x 20 min      =  4 x 1,200     =   4,800 s/yr
                                                            ---------
                                                            11,040 s/yr
seconds in a year  =  31,536,000
availability  =  1 - 11,040 / 31,536,000  =  0.99965

99.965% is 3.1 hours a year, and 6,240 / 11,040 = 57% of it is deploys, which are self-inflicted and vanish the moment there are two boxes to roll through. That number, and its gap from theory, is the reason a second web instance is coming.

Why the model says seven nines and reality says four

Add a second box and the naive math looks incredible. With two boxes, the service is down only when both are down at once, so if failures were independent:

one box unavailable, unplanned  =  4,800 / 31,536,000  =  1.52e-4
both at once, if independent     =  (1.52e-4)^2         =  2.3e-8   ->  0.99999998

That is seven nines, and nobody ever measures it, because no one runs long enough to observe an event that rare. Real multi-instance services land near 99.99%, and the whole gap is correlated failure: one bad deploy that goes to every machine, one availability zone losing power, one config push, one expired certificate. The model gives seven nines, reality gives four, and that difference is exactly what is worth engineering, starting with how we roll out the second instance.

Step 2 — Load balancer and a stateless-ish web tier

With the database on its own machine, the next move multiplies the web tier, and it is for availability, not speed. A load balancer is a machine (usually a managed service) that owns the public address, accepts every request, and forwards it to one of several identical web servers behind it. Because clients only ever talk to it, any server behind it can be replaced without a client noticing.

This rung is not forced by throughput

At 100k DAU the web tier needs 69 x 0.008 / 0.7 = 0.79 cores, a fraction of one machine, and it will not need more for years. So throughput is not the trigger. What forces the rung is the 6,240 seconds of annual deploy downtime plus the fact that with one instance, replacing it is a total outage. Availability, again, is the mover.

Sizing the fleet at 10M DAU

peak 6,944 QPS x 8 ms         =  55.6 cores of work
at rho 0.7                    =  55.6 / 0.7  =  79.4 cores
4-vCPU instances              =  79.4 / 4    =  19.9   ->  20
survive losing 1 of 3 AZs -> 2/3 of the fleet must carry all 20
                              =  20 / (2/3)  =  30 instances, 10 per AZ
cost  30 x 4 vCPU x $0.05/vCPU-hour x 730 hours/month  =  $4,380/month

The jump from 20 to 30 is the availability tax, and it is worth naming out loud in an interview. An availability zone is one independently powered building inside a region; providers give you three. If one goes dark, the remaining two thirds still have to carry all 20 machines’ worth of load, hence 20 / (2/3) = 30.

Three load-balancer properties worth knowing

• Health checks must test the dependency, not the process. A health check is a request the balancer sends every few seconds to decide whether an instance is fit for traffic. A /healthz that returns 200 just because the process is up keeps a dead instance in rotation while its database connection pool is exhausted. Check a real query, and cache that result for 1 second so the probes themselves do not become load.
• Least-outstanding-requests beats round robin when service times are heavy-tailed. Round robin feeds an instance stuck on a 2-second query at the same rate as a healthy one; least-outstanding-requests sends each request to whichever instance has the fewest in flight, noticing the stuck one within a single request.
• Connection count multiplies. Each instance keeps a connection pool (a set of reused open database connections). The pool is per instance, so 30 instances x 20 connections = 600 database connections against an optimum near 34, and each connection costs the database memory and scheduling whether busy or not (Database internals). Scaling the app tier degrades the database unless a connection pooler (a small proxy holding one modest set of real connections and multiplexing every instance onto it) sits between them. Introduce it here, where the multiplication first bites.

Step 3 — Read replicas, and the bug they introduce

A single database eventually stops being enough, and the fix comes with a catch. Read replicas add read capacity and hand us one specific correctness bug in exchange, so we cover both. The primary is the one database that accepts writes and is therefore the authoritative copy. A read replica continuously replays the primary’s write-ahead log to stay nearly identical, and answers reads only. Adding replicas multiplies read capacity and adds nothing to write capacity, because every replica has to apply every write anyway.

The measurement that forces replicas

The primary runs out of processor time. Watch an 8-core primary at 5 ms per request:

capacity at rho 0.7   =  8 x 0.7 / 0.005  =  1,120 peak req/s
1,120 / 3 = 373 avg QPS  ->  373 x 86,400 / 20  =  1,612,000 DAU

Read replicas are forced at 1.6M DAU, and the p99 signals it well before that. Service time on the box is 0.625 ms:

1.0M DAU   rho = 694 x 0.005 / 8    =  0.434    p99 = 4.6 x 0.625 / 0.566  =  5.1 ms
1.6M DAU   rho = 1,111 x 0.005 / 8  =  0.694    p99 = 4.6 x 0.625 / 0.306  =  9.4 ms

The p99 nearly doubled while CPU went from 43% to 69%: the cliff from the utilization table, showing up in production exactly as the formula warned.

Sizing the split

Reads are 90% of the database’s work, so moving them off the primary gives the largest gain. Almost all reads leave, but not quite all, and the exception is the whole story of this step. Read-your-writes is the guarantee that a user who just wrote something sees it on their very next read; those reads cannot go to a replica that may not have the write yet, so they stay on the primary.

primary after the split, per request
  writes                                       =  0.50 ms
  read-your-writes reads: 694 writes/s each dragging
    one 0.3 ms lookup back, over 6,944 req/s   =  0.03 ms
                                                  -------
                                                  0.53 ms
  budget it at 0.7 ms (margin for paths later declared consistency-sensitive)
primary capacity  =  8 x 0.7 / 0.0007  =  8,000 peak req/s  ->  11.5M DAU
replica capacity  =  8 x 0.7 / 0.0045  =  1,244 peak req/s each
replicas at 10M DAU  =  6,944 / 1,244  =  5.6   ->  6, plus 1 spare

The bug: a user posts and does not see it

Here is the correctness bug the split buys us. Replica lag is the gap between a write committing on the primary and becoming visible on a replica. A normal steady-state figure for asynchronous write-ahead-log shipping inside one datacenter is 40 ms, where asynchronous means the primary confirms the write to the user without waiting for any replica to have received it.

Let’s trace one posts row through one minute, with a replica 40 ms behind, and watch it go wrong:

t = 0 ms     POST /post/91  body = "hello wrold"   commits on the PRIMARY
                replica row  -> (not yet applied)
t = 12 ms    the follow-up GET load-balances onto the REPLICA, 40 ms behind
                returns the row as of t = -28 ms: body = ""  (empty draft)
t = 15 ms    the edit form renders with an EMPTY textarea
t = 40 ms    the replica applies the insert. Nobody is looking.
t = 900 ms   the user types "hello world" and submits -- but the form also
                resubmits every other field as rendered, i.e. before the insert
t = 905 ms   primary row -> title = "", tags = [], body = "hello world"

The typo is fixed and the title and tags are gone. Notice what actually happened: the user did not see stale data and shrug, the user computed a write from stale data, and the write landed on the authoritative copy. This is a lost-update bug, not a staleness bug: staleness heals itself in 40 ms; this does not heal at all, because the primary now holds the wrong value and no replica will ever contradict it. That is why we cannot just wait it out; we need a fix that changes where the read goes.

The exact fix: the LSN token

The pattern is simple to state: the write returns a marker, and the next read presents that marker to decide where it is safe to read from.

sequenceDiagram
    participant C as Client
    participant A as App
    participant P as Primary
    participant R as Replica
    C->>A: POST /comment
    A->>P: INSERT
    P-->>A: ok, wal_lsn = 0/3A1F0C8
    A-->>C: 201 + X-Read-Token: 0/3A1F0C8
    C->>A: GET /post (X-Read-Token: 0/3A1F0C8)
    A->>R: replay_lsn?
    R-->>A: 0/3A1E900 (behind)
    A->>P: SELECT (fall back to primary)
    P-->>A: rows, including the new comment
    A-->>C: 200

An LSN is a log sequence number: a monotonically increasing byte offset into the write-ahead log. Postgres prints the 64-bit value as two hex halves separated by a slash, so 0/3A1F0C8 is byte 60,944,584 of the log. Because the log is append-only, comparing two LSNs is exactly asking which write happened first. Walk the diagram with that in hand:

• The primary commits the INSERT and hands back the log position of that commit, a free byproduct of the write, not an extra query.
• The app hands that position to the client rather than storing it, because the app tier is stateless (step 6) and the next request may land on any instance. A header, cookie, or JSON field all work; it only has to survive the round trip.
• On the next read the client echoes the marker back. The app now knows the exact freshness this user requires: there is no global staleness bound, only a per-session one.
• The replica reports how far it has replayed. replay_lsn < read_token means “not caught up”, so the read falls back to the primary. The token does not make the read wait; it makes the read choose. With six replicas the app tries the others first.

The gap in that comparison converts to about 7 ms of lag ((60,944,584 - 60,942,592) bytes / 278 KB/s), well inside the 40 ms this replica normally runs at, and the token still correctly says no. A heuristic time window would have called the replica fresh and shipped the stale read. The exactness is the whole point.

The four fixes, priced against each other

The LSN token is one option. Here are all four, with what each actually costs at 10M DAU so you can defend a choice:

Three guarantee names an interviewer will want

Whichever fix you pick, you should be able to name what it promises. Three guarantees come up constantly:

All three are defined against eventual consistency, which is weaker than it sounds: it promises only that if writes stop, every replica eventually converges. It says nothing about when (40 ms and four minutes both satisfy it) and nothing about what a single reader sees along the way. So it permits the trace above, permits a second read to return an older value than the first, and permits a reply before its post. That is why eventual consistency gives none of the three guarantees, and why each is bought separately.

Why fixed time windows lose

One tempting fix, “just wait max_expected_lag”, fails, and it is worth knowing why. Lag is bimodal, not smooth. It sits near zero and jumps to minutes during a bulk load or an index build (Database internals). So a max_expected_lag of 500 ms is a bet you lose during exactly the events that matter. Eject a replica from the read pool when its lag exceeds the window; it is the only fix that degrades instead of breaking. The replica fleet now scales reads correctly, but it does so by paying for the same data many times, and that bill is what forces the next rung.

Step 4 — Cache, and the ceiling that forces it

The replica fleet works, but it pays for the same data seven times over, and that duplication is what forces the next rung. A cache is a small, very fast key-value store (almost always memory-resident, almost always Redis or Memcached) that holds recently asked answers under a key identifying the question. A read that finds its answer is a hit costing half a millisecond; a miss pays the full database cost behind it.

The measurement: seven copies of one hot set

Step 3 left us six replicas and a spare at 10M DAU. Each is a full copy and must hold the hot set in RAM to keep the 97% buffer hit rate the 18 ms query assumes. Add that up:

hot set at 10M DAU
  1e7 DAU x 2 posts/day x 400 B                 =  8 GB/day
  x 7-day hot window x 3 (rows + indexes)       =  168 GB
7 read copies (6 + spare)  =  7 x 168           =  1,176 GB of RAM, the same 168 GB seven times

the cache holds the hot 1% ONCE
  100,000 timelines x 20 posts x 400 B          =  0.8 GB
  it removes 71% of the reads (derived below), so 6 + 1 replicas -> 2 + 1
  RAM after  =  3 x 168 + 0.8                    =  505 GB      ->  2.3x less

See the difference in mechanism: replicas scale reads by duplicating the entire dataset; a cache scales reads by storing only the part actually read. The replica multiplier grows with traffic while the hot set does not, so this is what binds at 10M DAU. Replicas also hit a hard wall, since each must replay 100% of the write stream, but that arrives at ~30M DAU and belongs to step 9; we do not spend one measurement on two rungs.

Why the hot 1% is 71% of the reads

The cache only helps if a small slice of keys carries most of the reads. It does, and here is why. Reads on a social feed are Zipf-distributed: rank objects most- to least-requested, and the probability of a request landing on rank r is proportional to 1/r^s. A handful of objects take a large share and the rest form a long thin tail. Take exponent s = 1, the classic Zipf law measured for web-page popularity.

Probabilities must sum to one, so divide by their total, which over N objects is the harmonic number H_N = 1 + 1/2 + ... + 1/N. The hit rate of caching the top k is then H_k / H_N, and since H_n ~ ln(n) for large n:

hit rate  ~  ln(k) / ln(N)

(Checked against the exact H_k/H_N at k=1e5, N=1e7: 0.714 vs 0.724, 1.4% apart, far inside the error on “reads are Zipf-1”. Use the logs.)

N counts distinct cache keys, not rows in the database: the most common error with this formula.

N = 10,000,000 distinct timelines (one per DAU)
  cache k =   100,000  (1%)   ->  ln(1e5) / ln(1e7)  =  11.51 / 16.12  =  0.714
  cache k = 1,000,000  (10%)  ->  13.82 / 16.12                       =  0.857

Caching 1% of the keyspace removes 71% of the reads. But if you cache individual posts instead of whole timelines, N is no longer 10M timelines but the 140M-post 7-day corpus, and the same k = 100,000 gives 11.51 / 18.76 = 0.61, or 61%, not 71%, which is one extra replica. Same formula, same k, one machine of difference, and the only change was the definition of an object. The number the formula turns on is N, and it counts cache keys.

The curve is punishing at the top. A 99% hit rate needs ln(k)/ln(1e7) = 0.99, so k = e^15.96 = 8.5e6, which is 85% of the keyspace in RAM. The first 1% buys 71 points; the last point of hit rate costs the entire corpus. That is why “we’ll just raise the hit rate” is not a plan.

What the 71% actually buys

effective read latency
  hit    0.714 x 0.5 ms                    =  0.36 ms
  miss   0.286 x (0.5 + 18) ms             =  5.29 ms
                                              -------
                                              5.65 ms   vs 18.0 ms uncached  ->  3.2x

replicas needed  =  6 x (1 - 0.714)  =  1.7   ->  2, plus 1 spare

Two wiring patterns, and which is the safe default

Knowing the cache pays off, we have to wire it in, and the two options differ in more than code. Cache-aside (also lazy loading): the application owns the cache, and the cache does not know the database exists. On a miss the application queries the database and puts the answer back.

READ  timeline(u):
    v = cache.get("tl:" + u)          # 0.5 ms
    if v is not None: return v        # 71% of the time, done
    v = db.query(TIMELINE_SQL, u)     # 18 ms, on a replica
    cache.set("tl:" + u, v, ttl=300)  # 5 min, so a missed invalidation self-heals
    return v

WRITE post(u, body):
    db.insert(POST_SQL, u, body)      # the database commits FIRST
    cache.delete("tl:" + u)           # then DELETE, never SET

A TTL (time to live) is an expiry after which the cache forgets an entry whether or not anything changed. Invalidation is removing an entry because the data changed; a missed invalidation is one you forgot, which is why a TTL is worth keeping even when you invalidate correctly: it caps any mistake at five minutes. The two write-path lines matter:

• db.insert before cache.delete so the database is the only thing ever told the truth. If the process dies between them, the cache holds a stale entry that expires in 300 s, a bounded staleness bug, not a lost write.
• delete, not set, so you never write a value no reader asked for. Two concurrent writers doing set can leave the older value resident forever; two concurrent deletes cannot leave anything wrong.

Read-through inverts ownership: the application only calls the cache, and the cache library fetches on a miss. It is less code, and it is the wrong default here, because the cache now sits on the write path’s correctness boundary. A read-through cache that is up but wrong is indistinguishable from a wrong database, with no layer to fall back to. With cache-aside the worst failure is a latency spike; with read-through it can be a correctness bug. That is the trade we take cache-aside to avoid.

Three cache decisions that carry numbers

• Size for the miss path, not the hit path. At 71% you run 2 replicas; a cold cache instantly demands 6. Either keep 6 (and save nothing) or accept that a cache flush is an outage and warm the cache before taking traffic. This is the most common cache incident.
• The eviction policy is a working-set question. LRU (least recently used) is right when the hot set is stable; for a feed where yesterday’s post is dead, a TTL plus LRU beats either alone, because the TTL removes what is old and LRU removes what is unpopular.
• Invalidation races and versioned keys are derived in Database internals; cite them rather than re-derive.

The thundering herd the TTL creates

The TTL that protects us also sets a trap. The thundering herd (or cache stampede) is what happens when a popular key expires and every request for it misses at once, so many callers run the same expensive query to produce the same answer. Its cost comes straight from Little’s law: things in flight equals arrival rate times duration.

one hot key: 6.0% of reads = 375 /s, in flight during one 18 ms rebuild = 6.7 duplicate queries
mass expiry: top 100,000 keys warmed together share one TTL, expire together
  0.71 x 6,250 = 4,438 /s onto the miss path, x 18 ms = 80 concurrent queries against a 34-connection pool

Those are two different failures needing two different guards, which is exactly why the incident keeps recurring at teams that shipped only one of them:

• Single-flight fixes the first. The first caller to miss a key takes a lock on that key, runs the query, and every other caller for it waits on that one result: 6.7 duplicate queries become 1. It does nothing for the second case.
• TTL jitter fixes the second. Write ttl = 300 + random(0, 60) so a synchronized warm-up does not become a synchronized alarm clock, spreading 4,438 misses over a minute.
• Probabilistic early expiry covers both: refresh a key at random, with probability rising as its TTL approaches zero, for a few percent of extra queries all the time.

Step 5 — CDN, forced by bytes

Every rung so far was forced by processor time. This one is different: it is the first rung forced by raw bytes. A CDN is a fleet of rented servers spread across hundreds of cities. Each one is an edge holding copies of your static files, images, JavaScript, stylesheets, so a user fetches them from a machine ten milliseconds away instead of from your origin, the servers you actually run.

The machine count says it all

10M DAU x 5 page loads/day x 2 MB of assets  =  100 TB/day
average egress   =  100e12 B / 86,400 s  =  1.16 GB/s  x 8  =  9.3 Gbps
peak 3x                                                     =  27.8 Gbps
at 1 Gbps/machine, 50% usable            =  27.8 / 0.5      =  56 machines

Serving static bytes needs 56 machines; the entire application needs 30. That single comparison is the argument for a CDN. The 1 Gbps at 50% usable figure is the pessimistic end of the range: on 10 Gbps machines the same 27.8 Gbps needs only 6 machines. Both are defensible; the low end holds here because origin egress at this shape is limited by TLS termination and connection state on the same boxes serving the 5% of requests the CDN misses, not by the network card. Either way the real point stands: 100 TB/day is a bandwidth business you do not want to be in.

Latency: three round trips before the first byte

TTFB is time to first byte. RTT is round-trip time. Three round trips complete before the first byte: TCP connection, TLS handshake (transport layer security, the encryption every HTTPS page uses), and request-plus-response.

origin, 75 ms RTT   =  3 x 75  =  225 ms TTFB
edge,   10 ms RTT   =  3 x 10  =   30 ms      ->  7.5x faster

The bill is where the CDN earns its keep

Speed is nice; the bandwidth bill is what makes this rung non-negotiable. Origin egress is $0.05/GB, CDN egress $0.02. Offload is the fraction of bytes the edge serves without asking your origin. Monthly volume is 100,000 GB/day x 30 = 3,000,000 GB.

no CDN                    3,000,000 GB x 0.05                    =  $150,000/month
CDN at 95% offload  2,850,000 x 0.02 + 150,000 x 0.05           =  $ 64,500/month

$150,000 down to $64,500 is 2.3x, or $85,500/month saved. The 56 CDN-equivalent machines would cost 56 x 4 x $0.05 x 730 = $8,176/month, a rounding error against the egress saving. So the bandwidth is the argument; the machines are not.

Three things worth saying about CDNs

• Cache-key discipline is where CDNs actually fail. The edge decides whether two requests are “the same” from the URL plus whatever the Vary response header names. A Vary: Cookie on a static asset makes every distinct cookie a distinct object, so each user gets their own copy, the hit rate collapses, and the bill returns to $150,000. Serve static files from a path that never sets cookies.
• Long TTL plus content-hashed filenames removes invalidation entirely. Put a hash of the contents in the name (app.4f1c8e.js) and give it a one-year expiry: it can never be stale, because changing the contents changes the name, and it never has to be purged from hundreds of edges.
• The CDN is your first absorber of a DDoS (distributed denial-of-service, many machines flooding you at once). The edge fleet soaks the flood before it reaches your origin, and that is often the larger reason to have one.

Step 6 — Make the web tier stateless

Like the CDN, this rung is forced by the error budget, not processor time. A server is stateful when it remembers something between requests that no other server knows, typically a logged-in session held in its own memory. It is stateless when it remembers nothing, so any server can serve any request. A stateful web tier forces sticky routing: the balancer must send each user back to the same instance, because only that instance holds their session. Let’s price what that stickiness costs.

The error-budget calculation that kills sticky routing

An error budget puts a number on it: promise 99.9% availability and you promise no more than 0.1% of requests fail, and that failure allowance is a budget you get to spend. The budget is a volume, so it comes from the average; a replacement window is a worst case, so it comes from the peak.

monthly requests  =  2,315 avg QPS x 86,400 x 30    =  6.0e9
error budget      =  6.0e9 x 0.001                  =  6,000,000 requests

one instance replacement, sticky sessions
  1/30 of traffic fails for 60 s  =  6,944 / 30 x 60  =  13,888 requests  =  0.23% of budget
deploys: 30 instances x 2/week x 4.3 weeks  =  258 replacements/month
  258 x 13,888  =  3,583,000 requests  =  60% of the entire monthly error budget, on deploys

So sticky sessions do not fail the throughput test; they fail the error-budget test, and by a wide margin. Autoscaling, adding and removing instances as load changes, only makes it worse, because scaling from 30 down to 10 overnight evicts two-thirds of live sessions on purpose. The state has to move off the web tier.

Three places to put the session instead

If the session cannot live on the web server, where does it go? Three options, and they trade differently on revocation. Revocation is the ability to cancel a login immediately: a logout, a password change, a stolen laptop. A JWT (JSON web token) is a small blob of user facts the server signs cryptographically, so any server can verify it without a lookup.

Why the denylist stays tiny

The third option looks best but invites one objection, so let’s kill it with a number. That option needs a revocation list (denylist): tokens cancelled before their natural expiry. The worry is that it grows without bound. Little’s law ends the argument, because an entry only lives until the token it names would have expired anyway:

1,000 revocations/day, 15-minute (900 s) token lifetime
  arrival rate  =  1,000 / 86,400  =  0.0116 /s   ->   live entries  =  0.0116 x 900  =  10.4

Ten entries. A 15-minute access token with a denylist gives us both zero-lookup validation in the common case and instant revocation, and the denylist fits in a variable. Let’s watch one request use it.

One authenticated request, traced

Three token fields matter: sub (the user), jti (a unique token id so it can be revoked individually), exp (expiry). HMAC-SHA256 is a keyed signature: a hash of the payload mixed with a secret only your servers hold, so anyone with the key can check it and nobody without it can forge it.

cookie holds one signed token, ~400 B
  {"sub": "u_8814", "jti": "t_4f1c", "exp": 1735689600, "scope": "user"}
  + HMAC-SHA256(header + payload, server_key)

1. app recomputes the HMAC with the shared key    ~2 us CPU, 0 I/O   -> genuine, NOTHING looked up
2. app compares exp against the clock             expired -> 401
3. app checks jti against the in-process denylist dict lookup on ~10 entries (refreshed from Redis every 1 s)
4. request proceeds with sub = u_8814

Read those two steps together: step 1 is why this scales, because the signature proves your servers minted the token, so establishing identity costs no round trip and any instance can serve any request. Step 3 is why it is still revocable, because a logout writes t_4f1c into a tiny shared set, every instance picks it up within a second, and 15 minutes later the entry drops because the token would have expired anyway. The client’s long-lived refresh token, used once per user per 15 minutes rather than once per request, is the only part of authentication that touches a database.

The rest of the state has to leave too

The session is the hard case, but it is not the only state hiding on the web tier. Chase down the rest:

• Uploaded files go to object storage (a service that stores whole files under a name, reachable from every instance, such as Amazon S3), never a local disk, because a scale-in would delete user data with the instance.
• Background job state goes to the queue in step 8.
• Rate-limit counters go to the shared store; if each instance counts separately, thirty instances each allowing 100 requests is an effective limit of 3,000.

Step 7 — Multiple data centers

Running the same stack in two parts of the world sounds like a traffic problem. It is not; the hard part is the data. A region is a cloud provider’s cluster of datacenters in one geography, Northern Virginia, Frankfurt. Machines inside one region are milliseconds apart; machines in different regions are tens of milliseconds apart. Two independent things force a second region, and we must keep them separate, because they push on different parts of the design.

Forcing function one: latency for distant users

30% of 10M DAU sit 75 ms away from the single region. Loading a page costs six serial round trips: three for the first byte (as step 5 derived) and three more for the API calls the page then makes. Serial means each finishes before the next starts, so they add:

3 x 75 (TTFB) + 3 x 75 (API calls)  =  450 ms before render
same page in-region, 5 ms RTT        =  3 x 5 + 3 x 5  =  30 ms   ->  15x

Forcing function two: availability of a whole region

A whole region is itself a correlated failure domain. One power event, one bug in the provider’s control plane (the software that manages the machines, as opposed to the machines themselves), or one region-wide config push takes all of it at once. Two regions do not multiply out to seven nines, for the same reason two instances did not, but they remove the single largest correlated failure domain you have left.

Where the data lives is the whole problem

Steering traffic to a second region is a DNS setting. The genuinely hard decision is where the data lives.

flowchart TD
    U1(["EU user"]) -->|"geoDNS · 5 ms"| E["EU region<br/>app + cache + replicas"]
    U2(["US user"]) -->|"geoDNS · 5 ms"| W["US region<br/>app + cache + PRIMARY"]
    E -->|"reads: local replica"| ER[("EU replica")]
    E -->|"writes cross · 75 ms"| P[("US primary · writes")]
    W -->|"reads and writes stay local"| P
    P -.->|"async WAL · lag 75-200 ms"| ER
    ER -.->|"failover promotes this replica<br/>data loss = lag"| E

    style P fill:#1d3557,color:#fff

geoDNS, DNS that answers with a different address depending on where the query came from, puts each user 5 ms from an application tier. The two regions are deliberately not the same stack: the EU region has an app tier, a cache and replicas but no writable database; the US region has all that plus the primary, which is the authority. That asymmetry is the design:

• In the US, reads and writes stay local.
• In the EU, reads go to the local replica (5 ms).
• In the EU, writes cross the ocean to the primary (75 ms).

Because the workload is 90% reads, that asymmetry works in our favor: the average write penalty for an EU user is 0.1 x 75 ms = 7.5 ms. The dashed arrows are the things nobody waits for. Async WAL shipping carries committed log records to the EU replica without waiting for acknowledgement, so the replica trails by the round trip plus queueing, 75–200 ms in normal operation. That range is why the failover arrow is dashed: promoting the EU replica when the US region is gone is a decision you make in seconds, but it publishes a database missing everything in that gap.

Four data strategies, and when each is right

The single-primary layout above is one strategy of four. Here they are side by side, because the right one depends entirely on the write pattern. OLTP is online transaction processing: the ordinary read-write traffic of an application, where a 75 ms wait per write is fatal. A CRDT (conflict-free replicated data type) is a data structure whose independent copies can always be merged without a human choosing a winner; the alternative, last-writer-wins, resolves conflicts by discarding one edit.

How much a failover actually loses

Async replication has a price, and it comes due at failover. Take the middle of the 75–200 ms range, 150 ms. With asynchronous replication, a hard regional failover loses up to that much of committed writes:

writes in flight at failover  =  694 writes/s x 0.15 s  =  104 transactions

Whether 104 lost posts is acceptable is a product question. Whether 104 lost payments is acceptable is not, and that line decides which of the four strategies a real system picks.

Step 8 — Message queue, and the concurrency arithmetic

This rung is orthogonal to the last two; nothing in step 7 causes it. The arithmetic holds at 1M DAU exactly as at 10M, and if your product has one slow endpoint you should do this right after step 2. It sits here only because it is easiest to see once the fleet has a size. A message queue is a durable list: instead of doing slow work while the user waits, the web server appends a short job description and answers immediately, and separate machines called workers pull jobs off and do them. A thread is one unit of a server’s ability to have a request in progress, and a server has a fixed pool of them.

The forcing measurement is Little’s law

The trigger here is concurrency, not CPU, and Little’s law makes it visible. For any stable system, L = lambda x W: the number of items inside equals the arrival rate times the time each spends there. It needs no assumption about the distributions, because it is an accounting identity. Here L is threads held, lambda is requests per second, W is how long a request holds a thread. A photo post resizes into 5 variants at 200 ms each, so its W is a full second, while every other request’s W is the 8 ms from the workload block:

photo posts  =  10% of writes  =  694 x 0.1  =  69 /s,  work per post = 5 x 200 ms = 1.0 s
L, photo work        =  69 x 1.0        =  69 threads
L, everything else   =  6,944 x 0.008   =  56 threads

A thread is held for the whole duration of the work, so a request 125x slower costs 125x the concurrency at the same rate. That is how 1% of the traffic outweighs the other 99%. And the failure is not slowness, it is head-of-line blocking: the photo requests fill the thread pool and the 8 ms requests queue behind them the way one slow shopper holds up a checkout line, so the p99 of unrelated endpoints collapses.

What the queue buys back

With a queue, the web tier only performs an enqueue (appending the job description) and releases the thread immediately:

enqueue cost = 1 ms   ->   concurrency held = 69 x 0.001 = 0.07 threads

69 threads become 0.07. The second thing it buys is burst absorption. Provision workers at 2x peak (138 jobs/s) and hit them with a 10x burst for 60 s:

arrivals  =  690 x 60 = 41,400,  served = 138 x 60 = 8,280,  backlog = 33,120
drain rate = capacity - offered load = 138 - 69 = 69 /s,  drain time = 33,120 / 69 = 480 s = 8 minutes

Read the drain math carefully: drain time is set by the headroom, not the throughput, so a fleet provisioned at exactly 69/s has zero surplus and never drains. With headroom, a 10x burst becomes an 8-minute delay in thumbnail availability instead of a site outage.

Three properties of a queue to get right

• Consumer lag is the metric to watch, not throughput. Consumer lag is how far behind the workers are: the backlog, or better, how long it would take to clear. Throughput looks healthy right up to the moment the backlog grows without bound, because a saturated fleet runs flat out. Alert on backlog / drain_rate, which comes out in seconds.
• Handlers must be idempotent, because delivery is at-least-once. At-least-once means a job is delivered but maybe more than once: a worker that crashes after finishing but before acknowledging sees it again. Idempotent means running an operation twice leaves the same result as once. Key every job on a deterministic id so a duplicate resize overwrites the thumbnail rather than appending a second.
• Run a dead-letter queue with a size alarm. A dead-letter queue is where a job goes after failing too many times, so it stops being retried forever. Without one, a single poison message retrying endlessly is indistinguishable from a healthy queue on a throughput dashboard.

Step 9 — Sharding, last

We save the one move you cannot undo for last, which is why it comes with three forcing measurements and six cheaper alternatives to try first. To shard is to split the data itself across several databases, each holding a disjoint slice of rows; which database a row lives on is decided by its shard key, one chosen column. Sharding is the only move in this lesson that adds write capacity, and the only one you cannot cheaply undo.

Three measurements force sharding, and none is “it’s slow”

Notice what is missing from the list below: “the database is slow” never appears. Any one of these three is enough on its own.

1. REPLICA APPLY
   every replica replays 100% of the write stream, and apply is far less parallel than the write path
   single-threaded ceiling  =  1 / 0.002 ms  =  500 writes/s
   parallel apply (~4 streams)  =  4 x 500   =  2,000 writes/s
   the ladder crosses that at 30M DAU: 30e6 x 2 / 86,400 x 3 = 2,083 writes/s
   -> past this point replicas cannot keep up with the primary, no matter how many you run

2. WORKING SET
   posts at 100M DAU = 1e8 x 2 x 400 B = 80 GB/day, x 7-day window x 3 = 1,680 GB
   -> needs a 2 TB-RAM instance, the top of the SKU list

3. RESTORE TIME
   one year of posts = 80 x 365 = 29,200 GB, restore at 1 GB/s = 8.1 hours
   -> that is your recovery-time objective floor, and replicas do not move it
   sharded 16 ways = 29,200 / 16 = 30 min

The restore-time argument is the one operators find most convincing. A single 29 TB database has an 8-hour recovery-time objective no matter how many replicas you run, because of logical corruption: a bad migration or a buggy UPDATE that writes wrong values. That is a legitimate write, so every replica faithfully applies it. Replicas protect you from a machine dying; they do nothing about a mistake.

Try these six things first, in this order

Because sharding is irreversible, we exhaust the cheaper moves first, and in this order:

Sharding stays last because it is the only item you cannot undo cheaply. Shard-key selection, hot shards, hash vs range, the mod N reshuffle, and consistent hashing are all derived in Database internals and How consistent hashing works; we cite them rather than re-derive.

The operational shape comes down to three decisions

When you do shard, three choices decide whether it hurts once or forever:

• Fix a large number of logical shards up front. A logical shard is a bucket a row is assigned to permanently; a physical node is a machine, and many logical shards live on one. Route on hash(user_id) mod 1024, never mod 16, and keep a lookup table from logical shard to physical node. Adding a node then means editing that table and copying the logical shards you reassigned: about 6% of the data moves (64 / 1024), no row is ever rehashed, and no client needs to know the node count. Route on mod 16 directly and adding a node changes the destination of ~94% of keys, and that retrofit is a quarter-long project.

• Shard by user_id, not post_id. The dominant query is “this user’s timeline.” Sharding by post makes every timeline read a scatter-gather, one query fanned out to all 16 shards and merged, and a scatter-gather is only as fast as its slowest participant, so your p99 becomes the max over 16 shards. For the whole to be fast at the 99th percentile, each shard must be fast at 0.99 = p^16, i.e. p = 0.99937: a 99th-percentile promise on the fan-out is really a 99.94th-percentile demand on each shard.

• The migration is the project, not the schema. Five steps, each depending on the previous being finished:

  1. Dual-write every change to both old and new layouts so they stay in step.
  2. Backfill historical rows into the new layout, throttled over days so it does not swamp the primary.
  3. Verify with range checksums (hash each contiguous block of rows on both sides and compare) so you learn that the copies agree.
  4. Flip reads over one slice of traffic at a time.
  5. Only then stop dual-writing.

  Step 3 is the one teams skip and regret.

The architecture, assembled

All ten rungs now fit on one page, every box carrying the number that forced it. Read it top to bottom as the path a request takes; the dashed arrows are the things nobody waits for.

flowchart TD
    U(["Users"]) --> DNS["geoDNS"]
    DNS --> CDN["CDN edge<br/>95% offload · 30 ms TTFB"]
    CDN -->|"5% miss"| LB["Load balancer<br/>least-outstanding-requests"]
    LB --> W["Web tier · 30 x 4 vCPU<br/>stateless · rho 0.7"]
    W --> SS[("Session + rate limits<br/>Redis")]
    W --> C[("Cache · 1% of corpus<br/>71% hit rate")]
    W --> POOL["Connection pooler"]
    W --> Q[["Queue"]]
    Q --> WK["Workers<br/>resize · fan-out · email"]
    POOL --> PRI[("Primary<br/>writes + read-your-writes")]
    POOL --> REP[("Replicas x 2<br/>warm spare x 1")]
    PRI -.->|"async WAL"| REP
    WK --> PRI
    W --> OBJ[("Object storage")]
    CDN -.->|"origin pull"| OBJ
    ALL["Metrics · logs · traces<br/>sampled at 2.1%"] -.- W

    style PRI fill:#1d3557,color:#fff

Let’s walk one timeline request through it, box by box, each with the number that put it there:

• geoDNS answers with the nearest region, so the first packet travels 5 ms rather than 75 (step 7).
• The CDN edge serves the page shell, JavaScript and images: 95% offload, 30 ms TTFB against 225 ms from the origin (step 5). What it lacks it fetches by origin pull from object storage, which with content-hashed filenames and a one-year TTL happens effectively never.
• The remaining 5% miss is the only traffic reaching the load balancer, which picks the instance with the fewest outstanding requests (step 2).
• The web tier is 30 four-vCPU instances at rho 0.7: 20 for the load, 30 so losing an availability zone is uneventful. It is stateless, so the request carries its own identity: a signed token verified locally, checked against the ~10-entry denylist in the same Redis (step 6). Rate limits live there too, because 30 instances each counting to 100 is a limit of 3,000.
• Reads try the cache first: 1% of the corpus resident, 71% hit rate, 0.5 ms on a hit against 18 ms on a miss (step 4).
• Everything that misses goes through the connection pooler, without which 30 instances x 20 connections is 600 against an optimum of 34. Behind it, the primary takes writes plus the read-your-writes reads, and replicas x 2 plus a warm spare take everything else, fed by async WAL, the lag every consistency fix in step 3 is fighting.
• Anything holding a thread for more than a few milliseconds goes on the queue, where workers resize images, fan out writes and send email off the request path (step 8). They write to the primary, not a replica: they produce state.
• The observability plane holds metrics (pre-aggregated counters and timings), logs (one text record per event), and traces (every hop of a single request stitched together). Only the traces are sampled, at 2.1%, which keeps every error and slow request whole. The next section is why that percentage is the interesting number.

Observability, and the bill that is not where you think

One box on that diagram hides a bill most people size wrong by two orders of magnitude, so it is worth a careful look. Start with the first-order logging estimate everyone writes:

10M DAU, 200e6 requests/day, 1 KB structured log per request
  200e6 x 1,000 B = 200 GB/day  x $0.10/GB = $20/day = $600/month

$600/month against an app tier of $4,380/month makes logging 0.14x the compute it observes. (The one way this slips: mistake megabytes for gigabytes anywhere in the chain and the same inputs give 200,000 GB/day, a bill 137x the compute. Carry the raw byte count before converting.)

But 1 KB per request was never realistic, because a request does not emit one log line; every service it touches emits its own. Invert the arithmetic to find the crossover: for logging to cost what compute costs ($146/day, 1,460 GB/day), you need 1,460e9 / 200e6 = 7.3 KB per request, which is roughly thirty structured log lines, a normal request through a normal call graph. So across 20 services:

20 services x 1 KB = 20 KB/request  ->  200e6 x 20,000 B = 4,000 GB/day = $400/day = $12,000/month = 2.7x the app tier

The lesson is not “logging is expensive.” It is that a logging estimate has no meaning until you state bytes per request summed over the call graph, and that the fan-out, not the line size, is what crosses the compute bill. The fix is to record less without losing anything you would look at.

Head-based sampling decides at the moment a request starts whether to record it: keep everything that went wrong, and only a small random slice of the rest.

keep 100% of errors        =  200e6 x 0.001   =    200,000 /day
keep 100% of the slow tail =  200e6 x 0.01    =  2,000,000 /day
1% sample of the remainder =  197.8e6 x 0.01  =  1,978,000 /day
                                                 ---------
                                                 4,178,000 /day  =  2.1%
bill at 20 KB/request  =  0.021 x $400  =  $8.40/day  =  $252/month  =  0.06x the app tier

Head-based sampling that keeps every error and every p99 request costs 2.1% of the bill and loses nothing you would have looked at, taking observability from 2.7x the compute to 0.06x. Metrics and traces are different products: metrics are pre-aggregated, cheap and always on; traces are sampled, expensive and for the tail. Conflating them is how the $12,000 line appears in the first place.

Failure modes

Knowing the rung is only half of knowing the design; the other half is how each one fails in production. Read the detection column first, because most of these are invisible on the dashboard you would naturally build. A queue with an unbounded backlog shows perfect throughput; a thundering herd is a miss-rate spike with no traffic change; a connection-starved tier shows healthy CPU. The guards are cheap; noticing is the expensive part. An SLI in the third column is a service level indicator, a measurement chosen as a proxy for health.

Alternatives rejected

A good interview answer also knows the roads it did not take. Several other designs are reasonable, and each has something genuinely right about it. Each is ruled out here by a specific measurement at this scale, not by taste.

Summary

Every rung of the ladder is forced by one measurement. Here they all are, compressed to one line each so you can recall the chain under pressure:

Conclusion

The single most useful habit here is to size before you draw: turn a user count into a request rate, a request rate into cores, and cores into machines, and only then put boxes on a page. The M/M/1 result p99 = 4.6 S/(1-rho) is why 70% is the target utilization and why “20% CPU headroom” is not headroom. Read replicas, caches, and CDNs each scale reads a different way, by duplicating the dataset, by storing only the hot part, and by moving bytes to the edge, and each is forced by a different ceiling. Sharding, the one move you cannot cheaply undo, is worth delaying until a measurement forces it, and the two things a naive dashboard misses, correlated failure and unbounded queue backlog, are worth instrumenting before they bite.

One line to remember: never add a box until a number crosses a line, and always be able to name the number.

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications: the definitive treatment of replication, consistency guarantees, and partitioning behind steps 3, 7 and 9.
• Lee Breslau et al., “Web Caching and Zipf-like Distributions: Evidence and Implications” (INFOCOM 1999): the measurement behind the Zipf assumption used to size the cache.
• Marc Brooker, “Timeouts, retries, and backoff with jitter” (Amazon Builders’ Library): the retry-storm and jitter guards in the failure-modes table.
• Giuseppe DeCandia et al., “Dynamo: Amazon’s Highly Available Key-value Store” (SOSP 2007): eventual consistency and consistent hashing in a production system.
• Brendan Gregg, Systems Performance: utilization, queueing, and the USE method for the observability plane.

Next: Back-of-the-envelope estimation: the arithmetic underneath every number on this page.