01 — Scale From Zero To Millions Of Users

“You have one server and a thousand users. Walk me to ten million.”

This chapter takes one ordinary web application from a single machine to ten million daily users in ten steps, and at every step a specific measurement forces the next change.

You will meet the whole standard vocabulary of scaling — load balancer, read replica, cache, content delivery network, stateless tier, message queue, shard. Each one is defined in plain words at the moment it is first needed, so this chapter can be read having read nothing else.

When you finish you should be able to take a traffic figure someone hands you, turn it into a number of machines, and name the one measurement that forces the next architectural move.

The whole argument runs on eight lines of workload: how many people use the product each day, how many requests each of them makes, how much processor time one request costs in the application and in the database, and how many bytes it stores.

The output is an architecture with a number written under every box — thirty web machines, one primary database, two read replicas, a cache holding 0.8 GB, a queue and its workers. For each box you also get the measurement that forced it into existence. Everything in between is arithmetic, not taste.

Six words before the argument starts

Every figure in this chapter is quoted in these six terms, so they come first.

Two of those rows deserve a sentence more.

Latency and throughput are different quantities, and a system can be excellent at one and terrible at the other. A system that handles 10,000 requests per second where each takes two seconds has great throughput and terrible latency.

p99 is a way of reading a pile of measurements. Line the day’s requests up slowest-last, then read the value one percent from the slow end. A p99 of 5.1 ms means ninety-nine requests in a hundred finished inside 5.1 ms and one did not. This chapter quotes percentiles rather than averages because one request in a hundred being terrible is something users notice and an average hides.

How to read the rest

This is one architecture told eleven times, and every step is forced by a measurement, not by a preference.

A candidate who says “and then we add a cache” is reciting. A candidate who says “at 1.6M DAU the primary crosses 70% CPU and the p99 has already doubled, so the read path has to leave the primary” is designing. The difference is the whole round.

Two other chapters carry material this one leans on, and neither of them is required reading. The arithmetic style behind every estimate here is drilled in chapter 02. The database mechanism this chapter spends — B-trees, replication modes, isolation levels, shard keys — is derived in sql 03. Wherever one of their results is needed below, it is restated in a sentence first and linked second, so following the argument never requires leaving this page.

The workload, stated once

Eight numbers drive everything the chapter computes, and they are fixed here so that no later step can quietly invent a ninth.

Every figure about the workload comes from the block below. Every figure is quoted per DAU, because daily active users is the one number a product manager can actually give you.

The eight workload numbers

Read the block as the brief someone handed you. Nothing else about the product is known.

APP            a social feed: post, follow, read a home timeline
REQUESTS       20 per DAU per day, of which 10% are writes
PEAK           3x average          (stated, not derived -- defended below)
APP CPU        8 ms per request
DB CPU         5 ms per request, averaged over the mix
POST SIZE      400 B stored
FOLLOWS        300 per user, mean
COMPUTE        $0.05 per vCPU-hour

Hardware constants are a separate list, and you must not mix them up

The eight numbers above describe your users. Everything else this chapter spends describes the machines, and about fifteen more of those arrive later, each named on the line that first uses it. Five carry most of the chapter, so meet them now.

The other ten or so — page-read latencies, network-card throughput, instance ceilings, per-gigabyte prices — appear once each, where they are used.

Keep the two lists apart when you do this out loud. An interviewer will challenge a workload assumption and accept a hardware constant, so you want to know which one you just used.

Why the peak multiplier is 3x and not 2x

The PEAK line says the busiest second of the day carries three times the traffic of the average second. It matters because you buy machines for the busy second, not for the average one.

It is the only one of the eight that is an assumption rather than a measurement, and it multiplies every capacity number in the chapter, so say it out loud.

Qps worked end to end is explicit that the multiplier is stated, not derived. It gives 2x for smooth consumer traffic with one diurnal hump — one broad rise and fall across the waking day — and 3x for bursty or event-driven traffic.

This chapter takes 3x for two reasons. The feed is push-notified, and a notification fan-out — one event causing a message to be sent to many users at once — concentrates arrivals into minutes rather than spreading them across an evening. And the multiplier sizes headroom, meaning the spare capacity you hold above today’s load, where being wrong high costs money and being wrong low costs an outage.

Ch 03 sizes the same application at 2x, and its fleet is correspondingly 1.5x smaller. Neither is wrong; quoting one without naming it is.

Why this chapter’s read:write ratio disagrees with chapter 03

The REQUESTS line needs the same treatment, because the repo contains two different answers and you should know which you are holding.

• This chapter: 20 requests per user per day, 10% of them writes. That is 2 writes per user per day, and a 9:1 read:write ratio.
• Ch 03: 0.1 posts per user per day against 20 feed opens. That is a 200:1 ratio.

Neither is a typo and both are load-bearing, because they are load-bearing for different things. Every rung of this chapter’s ladder divides by 2 writes/day: the 694 peak writes/s at 10M DAU, the 8 GB/day hot set, the 140-million-object post corpus behind the cache hit rate, and the replica-apply wall at ~30M DAU. Ch 03’s 0.1 is what puts its design at 200:1 and forces it to fan out on write.

The two land on opposite sides of ch 03’s own ~10:1 threshold, so the architectures differ. This chapter never fans out on write; it caches computed timelines, which is what a 9:1 ratio asks for.

Say which user behaviour you assumed before you quote a ratio. A chatty feed and a quiet one are different products with the same diagram.

Where the 5 ms of database time goes

The 5 ms of database processor time is an average over a mix of cheap and expensive queries. Splitting it into its parts is what makes replicas and caches sizeable later, so do it now.

The request mix is the proportions in which the different kinds of request arrive: most requests do a few small lookups, one in five builds a timeline, one in ten writes something. The block multiplies each kind’s cost by how often it happens, and the three products sum to the 5 ms.

per request, averaged over the request mix
  3 point lookups x 0.3 ms                        =  0.9 ms
  0.2 timeline queries x 18 ms                    =  3.6 ms
  0.1 writes x 5 ms (row + 3 indexes + WAL)       =  0.5 ms
                                                     -------
                                                     5.0 ms
  of which reads                                  =  4.5 ms   -> 90%

Two terms in that block need naming, and both come from the row + 3 indexes + WAL note on the write line.

An index is a side structure the database maintains so it can find rows by a column’s value without reading the whole table. Keeping three of them current is why one logical write costs four physical ones.

WAL is the write-ahead log. Before a row is changed in place, the change is appended to the end of a log file and forced to disk, so a crash halfway through the real update can be replayed forward from the log instead of leaving a half-written page behind. That makes the log the fifth write inside the 5 ms.

The write-ahead log is also the stream that replicas consume. Every replication and read-your-writes mechanism later in this chapter is a mechanism about the write-ahead log, so it is worth having the definition solid now.

Where the 8 ms of application time goes

The application’s 8 ms splits the same way, and it is the number that sizes the entire web fleet, so it is the more important of the two. Same reading: cost per kind of work times how often that kind happens.

per request, averaged over the same mix
  parse, authenticate, route                      =  1.0 ms
  marshal 3 queries + deserialize 3 result sets   =  1.5 ms
  0.2 timeline assemblies x 20 ms of templating   =  4.0 ms
  serialize + gzip the response                   =  1.5 ms
                                                     -------
                                                     8.0 ms

Half the app tier is one endpoint, which is the same shape as the database’s 3.6-of-5.0 ms. That is not a coincidence: the timeline is the product, and every layer of this chapter is paying for it.

Where the 18 ms timeline query comes from

The 18 ms is derived rather than assumed. It is the only expensive thing in the system, and every later rung of the ladder is an attempt to avoid running it, so it is worth seeing how it is built.

Building one home timeline means finding the newest posts written by each of the 300 people a user follows and merging them into one list.

The database finds each of those 300 starting positions by walking down a B-tree index — a shallow tree whose leaves point at rows, so locating one row costs about four page reads instead of a scan of the whole table. A page is the fixed-size block, typically 8 KB, that a database reads and writes as a unit.

The block below turns that into time. Read it as two costs added together: waiting for disk, and burning processor.

merge the 20 newest posts from each of 300 followees
  300 index descents x 4 pages each               =  1,200 page reads
  at 97% buffer-cache hit, 0.1 us hit / 100 us miss
    1,200 x (0.97 x 0.1 + 0.03 x 100) us          =  3.7 ms of IO wait
  heap fetches, top-k merge, serialize                14.3 ms of CPU
                                                      -------
                                                      18.0 ms

The middle line of that block is a weighted average, and it is worth substituting by hand once:

cost of one page read, averaged over hits and misses
  hit    0.97 x 0.1 us   =  0.097 us
  miss   0.03 x 100 us   =  3.000 us
                            --------
  per page                =  3.097 us
  x 1,200 pages           =  3,716 us  =  3.7 ms

Two constants in that derivation come from elsewhere and are worth stating in words.

Four pages per B-tree descent is derived in B trees why a lookup is four page reads. It holds because a tree with a few hundred children per node reaches billions of rows in four levels.

0.1 microseconds against 100 microseconds is the difference between finding a page already in the database’s buffer cache — the slice of memory in which it keeps recently used pages — and having to fetch it from the solid-state disk. It is priced in Caching layers and invalidation.

Hold on to that 1,000x ratio. It reappears at every layer of this chapter, because every layer is the same bet: keep the working set one tier up.

The traffic ladder

The table below turns each user count into a request rate, and every later step of the chapter reads its numbers off it.

Every row is the same two operations. Divide the day’s requests by the number of seconds in a day to get the average, then multiply by 3 to get the peak. Work the 10M row once and the other five are the same:

requests/day  =  10,000,000 DAU x 20 requests  =  200,000,000 /day
avg QPS       =  200,000,000 / 86,400 s        =      2,315 /s
peak QPS      =  2,315 x 3                     =      6,944 /s
reads/s peak  =  6,944 x 0.90                  =      6,250 /s
writes/s peak =  6,944 x 0.10                  =        694 /s

Reads and writes are both listed at peak, because the moment they are compared to each other — or to anything’s capacity — they have to be on the same footing.

Two conventions, stated before any of it is spent

Every rate in this chapter carries two labels, because a rate without them is the most common way a capacity calculation goes wrong by a factor of three. Both are bookkeeping rules, and skipping them produces a fleet that is exactly one surprise away from a queue.

The first label says whether a rate is the peak or the average.

Peak sizes anything with a queue in front of it — processor cores, replicas, connections, workers — because those things have to survive the busiest second.

Average sizes anything that accumulates — stored bytes, egress bills, error budgets — because those things only care about the total over a month.

Every rate below is peak unless the line says avg. A comparison between one of each is a bug, not a shortcut: the 3x factor is the only thing separating them, so mixing them silently is a 3x error in whichever direction flatters the design.

The second label says whether a rate is offered load or service capacity.

Offered load is what the workload asks for. 6,944 peak req/s is one.

Service capacity is what a machine supplies. 4 x 0.7 / 0.008 = 350 req/s is one, and it reads left to right as: 4 processor cores — a vCPU is one virtual core as a cloud provider sells it — run at a 70% target rather than flat out, with each request costing 8 ms, which is 0.008 s, of processor time.

Then the division tells you which question you just answered:

Never divide two offered loads and call the result a utilization. That is the error the rule exists to stop.

State whether a rate is offered load or service capacity at the moment you derive it, and never divide one by the other without saying which is which.

A fleet sized so that capacity exactly equals peak offered load is running at 100% utilization — written rho = 1 below — and the next section is about what that costs.

The one law that forces most of the ladder

A single equation decides how busy you are allowed to let a machine get, and it is the reason seven of the ten rungs below exist at all. Utilization — the fraction of the time a machine is busy rather than idle — is not a slope, it is a cliff, and every “we’re at 80% CPU, we’re fine” incident is this equation.

The model, in three assumptions

Model a tier as an M/M/1 queue. That name is Kendall notation, a shorthand for the assumptions a queueing model makes, and each symbol is one fact.

• The first M says arrivals are Markovian, which here means Poisson: requests arrive independently of one another. The gap between two of them is exponentially distributed, so how long you have already waited tells you nothing about when the next one comes.
• The second M says service is Markovian too: the time to handle a request is drawn from the same kind of distribution.
• The 1 says there is one server.

Two symbols carry the arithmetic. S is the mean service time, meaning how long one request occupies the server once it starts, with no waiting counted in it. lambda is the arrival rate.

Utilization is then a rate multiplied by a duration:

rho  =  lambda x S / c        (c servers; c = 1 for M/M/1)

rho has no units, because a rate times a duration cancels out. Read it as the fraction of the time a server is busy, which is also the mean number of requests in service.

Substitute the 8-vCPU database primary at 1M DAU, where the ladder gives 694 peak requests per second and each costs 5 ms — that is 0.005 s — of database processor time:

rho  =  694 /s  x  0.005 s  /  8 cores  =  0.434

That 0.434 is the figure step 3 uses.

From utilization to p99

The one result to memorize: for M/M/1 the response time — the time waiting in the queue plus the time being served, which is what the user actually experiences — is itself exponentially distributed, with mean S/(1-rho).

That single fact gives you every percentile, because an exponential distribution with mean m has its q-th quantile at m x ln(1/(1-q)). A q-th quantile is just the value that a fraction q of the samples fall below, so the p99 is the quantile at q = 0.99.

Substitute q = 0.99 into ln(1/(1-q)):

ln(1 / (1 - 0.99))  =  ln(1 / 0.01)  =  ln(100)  =  4.605

So the two formulas the rest of the chapter uses are:

mean   =  S / (1 - rho)
p99    =  ln(100) x S / (1 - rho)  =  4.6 x S / (1 - rho)

The same formula, worked on real numbers

Take the same database primary. S is not the 5 ms of processor time a request costs — it is how long the request occupies the tier, and the tier has 8 cores working in parallel:

requests the box finishes flat out  =  8 cores / 0.005 s  =  1,600 /s
S = one over that                   =  1 / 1,600 s        =  0.625 ms

Now put S = 0.625 ms and rho = 0.434 into the two formulas:

1 - rho  =  1 - 0.434              =  0.566
mean     =  0.625 / 0.566          =  1.10 ms
p99      =  4.6 x 1.10             =  5.1 ms

Notice that S appears once and (1-rho) appears once. Making the code faster and taking load off the tier are the same lever, and only one of them requires a deploy.

What the cliff looks like

The table below evaluates that formula across the range of utilizations you might actually run at. Read the third column as “the p99 is this many service times”, and read the last column as “this much extra traffic doubles your p99” — the last one is what should change your behaviour.

The last column is the only one that needs deriving. Ask which higher utilization rho' would double the p99. Doubling the p99 means doubling the queue multiplier, so:

1/(1 - rho')  =  2 / (1 - rho)      ->   rho'  =  (1 + rho) / 2

traffic increase that gets you there  =  rho'/rho - 1

at rho = 0.9:   rho' = (1 + 0.9)/2 = 0.95
                0.95 / 0.9 - 1     = 0.056  =  +5.6%
at rho = 0.5:   rho' = 0.75
                0.75 / 0.5 - 1     = 0.50   =  +50%

At 50% utilization it takes a 50% traffic increase to double your p99. At 90% it takes 5.6%, which is the difference between a Tuesday and a Wednesday.

That is why every capacity number in this chapter targets rho <= 0.7, and why “we still have 20% CPU headroom” is not a true statement about headroom.

The formula as code

The two functions below turn those formulas into code you can run: the first gives a p99 from a service time and a utilization, the second gives a core count from a request rate.

The docstring of the first one warns against using it to promise an SLO, a service level objective: a published target such as “p99 under 100 ms”, which you are held to and which therefore needs measurement rather than a model.

There are two honest caveats worth volunteering before an interviewer raises them. Treating c cores as a single server that runs c times faster is optimistic, because a real bank of c separate servers — the M/M/c model — queues more than one fast server does. And real arrivals are burstier than Poisson, since users act in correlated waves rather than independently. Both errors point the same way, so a measured p99 is worse than this table, never better.

def p99_ms(service_ms: float, utilization: float) -> float:
    """M/M/1 p99 response time. The 4.6 is ln(100).

    Optimistic on two counts: c cores modelled as one c-times-faster server
    queue less than c real servers do, and real arrivals are burstier than
    Poisson. Use it to find the cliff, not to promise an SLO.
    """
    if not 0 <= utilization < 1:
        raise ValueError("utilization must be in [0, 1)")
    return 4.6 * service_ms / (1 - utilization)


def cores_needed(peak_qps: float, service_ms: float, target_util: float = 0.7) -> float:
    """Cores to hold `peak_qps` at `target_util`. 0.7 is the default because
    that is where a 21% traffic surprise merely doubles p99 instead of
    taking the tier down."""
    return peak_qps * (service_ms / 1000.0) / target_util

Those two functions produce three of this chapter’s headline numbers, and it is worth running them once so the later sections read as arithmetic rather than as claims:

# Step 3. The 8-vCPU primary, service time 0.625 ms, at 1.0M and 1.6M DAU.
print(round(p99_ms(0.625, 0.434), 1))   # 5.1 ms  <- comfortable
print(round(p99_ms(0.625, 0.694), 1))   # 9.4 ms  <- p99 nearly doubled for a
                                        #            26-point move in CPU

# Step 2. Sizing the web tier at 10M DAU: 6,944 peak QPS at 8 ms each.
print(round(cores_needed(6944, 8), 1))        # 79.4 cores
print(round(cores_needed(6944, 8) / 4, 2))    # 19.84 -> 20 four-vCPU instances

# "Your p99 is 400 ms and CPU is at 80%." No code is slow; S is 17 ms.
print(round(p99_ms(17, 0.8)))                 # 391 ms

The last call is the one to internalize. It answers the “your p99 is 400 ms and CPU is at 80%” question at the end of this chapter without profiling anything: a 17 ms service time at 80% utilization is a 391 ms p99, so an engineer hunting for a slow function is hunting for something that does not exist.

The ladder

Here is the map: ten rungs in the order they are forced, each labelled with the measurement that forces it. Read the diagram as a sequence in time — you climb it as traffic grows, and you never skip a rung because the rung above is more interesting.

flowchart TD
    S0["0 · Single box<br/>app + DB + files<br/>up to ~100k DAU"]
    S1["1 · Split the DB<br/>a RAM ceiling, not a CPU one"]
    S2["2 · Load balancer + N web<br/>forced by availability"]
    S3["3 · Read replicas<br/>forced at 1.6M DAU"]
    S4["4 · Cache<br/>forced by 7 copies of one hot set"]
    S5["5 · CDN<br/>forced by 28 Gbps of static bytes"]
    S6["6 · Stateless tier + session store<br/>forced by the error budget"]
    S7["7 · Multi-region<br/>Europe pays 450 ms"]
    S8["8 · Message queue<br/>forced by 70 threads for 1% of traffic"]
    S9["9 · Shard<br/>forced at ~30M DAU by replica apply"]

    S0 --> S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 --> S9

    style S0 fill:#1d3557,color:#fff
    style S3 fill:#2d6a4f,color:#fff
    style S4 fill:#2d6a4f,color:#fff
    style S5 fill:#bc6c25,color:#fff
    style S6 fill:#bc6c25,color:#fff
    style S8 fill:#40916c,color:#fff
    style S9 fill:#9d0208,color:#fff

One key covers this diagram and the two later ones in the chapter, so it is stated once here. On the ladder the colour describes the rung; in the later diagrams it describes the box that rung produced.

Two of those rows exist because the obvious four-colour version of this key does not survive contact with the diagrams. Green cannot mean “takes load off the request path” and also mean “read capacity”, because a queue does the first and none of the second — so the queue gets its own shade and green means read capacity throughout. And orange cannot mean “raw bytes”, because raw bytes are throughput; what steps 5 and 6 have in common is that neither was forced by processor time, which is the wording step 5 itself uses.

Every rung in one sentence

Each of these gets a full step below. If a word on the diagram meant nothing to you, its definition is on this page.

Three words the diagram spends before defining

Availability is the fraction of the time the service is actually answering, rather than failing or unreachable. It is conventionally quoted as a run of nines: 99.9% is about 8.8 hours a year of not answering, and 99.99% about 52 minutes.

Availability is not the same quantity as throughput and it is not bought the same way. Step 1 prices a single box at 99.965% and shows that 57% of that loss is deploys, which no amount of extra capacity removes. It is the forcing function for rung 2 and half of rung 7.

Rung 1 says “a RAM ceiling, not a CPU one.” On one box, the application’s heap — the memory it allocates for its own working data — and the database’s page cache compete for the same 16 GB. The loser is whichever one you notice second.

Rung 7 says “Europe pays 450 ms.” That is three serial round trips to fetch the page plus three more for the API calls the page needs, at 75 ms each. An API is one service’s programmatic interface, the thing another piece of code calls. No amount of extra capacity in one region removes those milliseconds; only a second region does.

Step 0 — One box, and how far it actually goes

How much traffic does a single ordinary machine really carry? Far more than most people guess — and knowing the number is what stops you from over-designing.

The setup is one machine with 4 virtual processor cores and 16 GB of memory, running the web application, the database and the static files together.

How much the box carries

Say its capacity out loud before you say anything else. The block prices the same box three ways: as if it only ran the app, as if it only ran the database, and as it actually is, running both.

app tier   =  4 x 0.7 / 0.008  =  350 req/s at rho 0.7
db tier    =  4 x 0.7 / 0.005  =  560 req/s at rho 0.7
combined on one box, 8 ms app + 5 ms db = 13 ms of CPU per request
           =  4 x 0.7 / 0.013  =  215 req/s

Each line reads the same way: 4 cores, run at a 70% target, divided by the seconds of processor time one request costs. The combined line is the real one, because both tenants are on the same 4 cores and their costs add.

Now walk 215 peak requests per second back to a user count, undoing the two operations the traffic ladder applied:

peak -> average    =  215 / 3            =  72 avg req/s
average -> per day =  72 x 86,400        =  6,220,800 requests/day
per day -> DAU     =  6,220,800 / 20     =  311,000 DAU on one box

Say that number early, because it kills the reflex to over-design. At 100k DAU the ladder offers this box 69 peak requests per second against a 215 capacity, so it is running at 69 / 215 x 0.7 = 22% utilization — and the correct architecture is one machine and a backup script.

Memory is what breaks first, not CPU

The two tenants of that machine want opposite things from the same 16 GB. The application wants heap for the objects it builds while serving a request. The database wants page cache, the region of memory in which it keeps recently used disk pages so it does not have to read them again.

Caching layers and invalidation puts the database’s appetite at 25-80% of the machine’s memory depending on the engine.

What has to fit in that page cache is the working set — the portion of the data that is actually being read at any moment. The block below traces how long the working set stays inside 16 GB at 100k DAU. Read it top down: three data sets, then what is left for them, then the date they overflow.

posts        100k DAU x 2 posts/day x 400 B      =    80 MB/day
follows      100k users x 300 x 16 B x 3 (index) =   1.44 GB
users        100k x 1 KB                         =   0.1 GB

available page cache = 16 GB - 4 (app heap) - 2 (OS)  =  10 GB
  of which follows + users already occupy          =   1.54 GB
  left for posts                                   =   8.46 GB
posts cross it after 8.46 / 0.08                   =   106 days

The follows line is the one worth unpacking: one follow edge is two 8-byte identifiers, so 16 B, and the x 3 is the row itself plus the two index entries the database keeps so it can answer “who does X follow” and “who follows X”. Follows and users do not grow with time in this window; posts do, which is why only posts get a per-day rate.

Subtract the follows and users before you divide. They are resident from day one. A reader who divides the whole 10 GB by 80 MB/day gets 125 days instead of 106 — 18% optimistic about the only deadline in this section.

What crossing that line costs

The consequence is not gradual, because a page found in memory costs 0.1 microseconds and a page fetched from disk costs 100 — one thousand times more.

The table prices the same 1,200-page timeline query at four different buffer hit rates, where the hit rate is the fraction of page requests answered out of memory. The first row shows the arithmetic; every other row is the same expression with a different pair of weights.

A 9-point drop in cache hit rate is a 9x latency increase. Nothing about the query changed; the working set left RAM.

Step 1 — Split the database off

When the 106-day deadline from step 0 arrives, the first machine to add is a second one for the database rather than a second one for the application.

The change is small: two machines, a web box and a database box, where before there was one. Nothing is faster on the day you do it. What you bought instead is four things, and the right-hand column is the number that makes each one real rather than a slogan.

The cost: every query now crosses a network

Name this one unprompted. On one box a query was a function call. Now it is a message to another machine and an answer coming back, which is 0.5 ms, and the timeline path issues 3 queries:

added latency  =  3 x 0.5 ms  =  1.5 ms per request
as a fraction of the 18 ms timeline query  =  1.5 / 18  =  8.3%

Paying 8.3% more on the timeline query to get all four of those benefits is fine.

The version that is not fine involves an ORM, an object-relational mapper: the library that lets application code treat database rows as ordinary objects. Left to itself, an ORM will lazy-load — fetch each related row only at the moment the code touches it.

So asking for 300 followees one at a time becomes 300 separate messages:

300 followees, fetched one at a time  =  300 x 0.5 ms  =  150 ms of pure round trip

That pattern is called an N+1 query: one query to get the list, then one more for each of its N members. It was invisible on one box and is now the entire latency budget — 150 ms of network against an 18 ms query. That regression is the single most common consequence of this step, and it is worth saying before the interviewer asks.

Vertical vs horizontal, with the ceilings

There are two ways to add capacity, and each hits a different ceiling.

Vertical scaling means replacing a machine with a bigger machine. Horizontal scaling means adding more machines of the same size.

Do not reflexively go horizontal. Vertical is one line of configuration and its range is enormous — the block below runs the same capacity formula on the smallest and the largest general-purpose instance you can rent:

one 4-vCPU box      =  4 x 0.7 / 0.008    =    350 req/s
one 192-vCPU box    =  192 x 0.7 / 0.008  = 16,800 req/s
ratio                                     =  16,800 / 350  ->  48x

The table compares the two on the five things that decide between them. The bolded row is the one that actually decides it, and it is not the throughput row.

Three terms in that table need naming.

A failure domain is a set of things that all go down together. A single machine is one: when it stops, everything stops. A failover is the act of promoting a standby machine when the live one dies. Availability is defined on the ladder above.

The ceiling that binds is availability, not throughput. Price the single box honestly. The block counts every second of the year the box is not answering, from the two sources that produce them, and turns the total into a fraction:

planned:   2 deploys/week x 60 s        =  104 x 60      =   6,240 s/yr
unplanned: 4 incidents/yr x 20 min      =  4 x 1,200     =   4,800 s/yr
                                                            ---------
                                                            11,040 s/yr
seconds in a year                       =  365 x 86,400  =  31,536,000
availability  =  1 - 11,040 / 31,536,000  =  0.99965

99.965% is 3.1 hours a year, and 6,240 / 11,040 = 57% of it is deploys, which are entirely self-inflicted and disappear the moment there are two boxes.

Why the model says seven nines and reality says four

With two boxes, the service is down only when both are down at once. If their failures were independent, the two probabilities would multiply:

one box unavailable, unplanned only  =  4,800 / 31,536,000  =  1.52e-4
both at once, if independent         =  (1.52e-4)^2         =  2.3e-8
availability                         =  1 - 2.3e-8          =  0.999999977

That is “seven nines” — 99.99999% available, counting the nines after the decimal point. Nobody measures seven nines, because no one runs long enough to observe an event that rare.

Real multi-instance services land near 99.99%, and the whole gap is correlated failure: failures that are not independent at all. One bad deploy that goes to every machine. One availability zone losing power. One config push. One expired certificate.

Say that out loud: the model gives seven nines and reality gives four, and the difference is the thing worth engineering.

Step 2 — Load balancer and a stateless-ish web tier

With the database on its own machine, the next move multiplies the web tier — and the reason is availability, not speed.

A load balancer is a machine — usually a managed service — that owns the public address, accepts every incoming request, and forwards it to one of several identical web servers behind it. Because clients only ever talk to it, any server behind it can be taken away and replaced without a client noticing.

This rung is not forced by throughput

At 100k DAU the web tier needs 69 x 0.008 / 0.7 = 0.79 cores, which is a fraction of one machine. Throughput is not the problem and will not be for years.

What forces the rung is the 6,240 seconds of annual deploy downtime priced above, plus the fact that with one instance, replacing that instance is a total outage.

Sizing the fleet at 10M DAU

Size it at 10M DAU instead, where the shape of the fleet is visible. Read the block as four steps: how much processor work arrives, how many cores that needs at a 70% target, how many machines that is, and how many more you need so that losing a building is uneventful.

peak 6,944 QPS x 8 ms         =  55.6 cores of work
at rho 0.7                    =  55.6 / 0.7  =  79.4 cores
4-vCPU instances              =  79.4 / 4    =  19.9   ->  20
survive the loss of 1 of 3 AZs -- availability zones, independently
  powered buildings in one region -- so 2/3 of the fleet must carry 20
                              =  20 / (2/3)  =  30 instances, 10 per AZ
cost  30 machines x 4 vCPU x $0.05/vCPU-hour x 730 hours in a month
                              =  30 x 4 x 0.05 x 730  =  $4,380/month

The jump from 20 to 30 is the availability tax. An availability zone is one independently powered building inside a region, and providers give you three. If one goes dark you are left with two thirds of your machines, and those two thirds still have to carry all 20 machines’ worth of load — hence 20 / (2/3) = 30.

Three load-balancer properties worth knowing

Each of these comes up in almost every interview, and each is a mechanism rather than an opinion.

• Health checks must test the dependency, not the process. A health check is a request the balancer sends every few seconds to decide whether an instance is still fit to receive traffic. A /healthz endpoint that returns HTTP 200 merely because the process is running will keep answering while the database connection pool behind it is exhausted, so a dead instance stays in rotation. Check a real query instead, and cache that result for 1 second so that N_instances balancers probing once a second do not become load in their own right.
• Least-outstanding-requests beats round robin when service times are heavy-tailed, meaning a small fraction of requests take far longer than the rest. Round robin hands each instance the next request in strict rotation, so it keeps feeding an instance that is stuck on a 2-second query at exactly the same rate as a healthy one. Least-outstanding-requests sends each request to whichever instance currently has the fewest still in flight, so it notices the stuck one within a single request.
• Connection count multiplies. Each application instance keeps a connection pool: a set of already-open database connections it reuses, because opening one per request is expensive. The trouble is that the pool is per instance, so 30 instances x 20 connections = 600 database connections against an optimum near 34. Each connection costs the database memory and scheduling whether or not it is doing anything, which is the exact failure Connection pooling matters more than people expect prices. Scaling the app tier degrades the database unless a pooler sits between them. A connection pooler is a small proxy that holds one modest set of real database connections and multiplexes every instance’s traffic onto it. This is the step where the multiplication first bites, so introduce the pooler here.

Step 3 — Read replicas, and the bug they introduce

Sooner or later a single database stops being enough. Read replicas fix that, and they introduce a specific correctness bug in exchange.

The vocabulary first. The primary is the one database that accepts writes, and is therefore the authoritative copy of the data. A read replica is a second database that continuously replays the primary’s write-ahead log to stay nearly identical, and answers read queries only.

Adding replicas multiplies read capacity and adds nothing at all to write capacity, because every replica has to apply every write anyway.

The measurement that forces replicas

The primary runs out of processor time. Take an 8-core primary spending 5 ms of processor time per request, and walk its capacity back to a user count exactly as step 0 did:

capacity at rho 0.7   =  8 x 0.7 / 0.005  =  1,120 peak req/s
peak -> average       =  1,120 / 3        =    373 avg QPS
average -> DAU        =  373 x 86,400 / 20  =  1,612,000 DAU

Read replicas are forced at 1.6M DAU, and the p99 warned you well before that.

To see the warning, put the same box through the queueing formula at two traffic levels. Service time on an 8-vCPU box is 1000 / (8 / 0.005) = 0.625 ms, the number derived in the queueing section:

1.0M DAU   rho = 694 x 0.005 / 8    =  0.434    p99 = 4.6 x 0.625 / 0.566  =  5.1 ms
1.6M DAU   rho = 1,111 x 0.005 / 8  =  0.694    p99 = 4.6 x 0.625 / 0.306  =  9.4 ms

The p99 nearly doubled while CPU went from 43% to 69%. That is the cliff from the utilization table showing up in production: the two rows differ by 26 points of CPU and a factor of two in what users feel.

Sizing the split

Reads are 90% of the database’s work, so moving them off the primary is the highest-leverage move available.

Almost all reads leave, but not quite all. Read-your-writes is the guarantee that a user who has just written something sees it on their very next read. Those reads cannot go to a replica, because the replica may not have the write yet, so they are the small residue that stays on the primary.

The block below sizes what stays behind and what leaves. Read the top half as “how much work per request is still on the primary”, and the bottom half as the machine counts that fall out of it.

primary after the split, per request
  writes                                       =  0.50 ms
  read-your-writes reads: 694 writes/s each
    dragging one 0.3 ms lookup back to the
    primary, spread over 6,944 req/s
    = 694 x 0.3 / 6,944                        =  0.03 ms
                                                  -------
                                                  0.53 ms
  budget it at 0.7 ms -- a 32% margin for the
  paths you will later decide must not be stale
primary capacity  =  8 x 0.7 / 0.0007  =  8,000 peak req/s  ->  11.5M DAU
   (at the measured 0.53 ms it is 10,600 req/s, or 15.2M DAU; quote the
    budgeted number, because the margin is what absorbs the next endpoint
    someone declares consistency-sensitive)
replica capacity  =  8 x 0.7 / 0.0045  =  1,244 peak req/s each
replicas at 10M DAU  =  6,944 / 1,244  =  5.6   ->  6, plus 1 spare

The bug: a user posts a comment and does not see it

Watching the read-your-writes failure end to end, as a user experiences it, is what turns “replicas are eventually consistent” from a phrase into a decision — so watch it before pricing the fixes.

Replica lag is the gap between a write committing on the primary and the same write becoming visible on a replica. Its mechanism is derived in Replica lag is a correctness problem, which is worth reading for depth but is not needed to follow what happens next.

The shape of the bug is the same whether the object is a comment or a post. Take a post, because it has more fields to lose.

The trace below follows one row of the posts table through one minute of a user’s life, with a replica whose apply is 40 ms behind. That 40 ms is a normal steady-state figure for asynchronous write-ahead-log shipping inside one datacenter, where asynchronous means the primary confirms the write to the user without waiting for any replica to have received it.

Read the trace as a clock. The thing to watch is not the stale read at t = 12 ms — it is what the user does with it at t = 900 ms. Keep the 40 ms in mind too, because a later paragraph calls this lag bimodal.

t = 0 ms     POST /post/91  body = "hello wrold"     commits on the PRIMARY
                primary row  -> "hello wrold"
                replica row  -> (not yet applied)
t = 8 ms     302 redirect to GET /post/91/edit
t = 12 ms    the GET load-balances onto a REPLICA, 40 ms behind
                returns the row as of t = -28 ms: body = ""   (empty draft)
t = 15 ms    the edit form renders with an EMPTY textarea
t = 40 ms    the replica applies the insert. Nobody is looking.
t = 900 ms   the user types "hello world" into the empty box and submits
                PUT /post/91  body = "hello world"
                -- but the form also submitted every other field as it was
                   rendered, i.e. as they were BEFORE the insert
t = 905 ms   primary row -> title = "", tags = [], body = "hello world"

The typo is fixed and the title and tags are gone.

The user did not see stale data and shrug. The user computed a write from stale data, and the write landed on the authoritative copy.

That is why the sentence to say is this is a lost-update bug, not a staleness bug. Staleness heals itself in 40 ms. This does not heal at all, because the primary now holds the wrong value and no replica will ever contradict it.

Fix one, traced: the LSN token

There are four fixes, priced in a table below. The sequence diagram traces the most exact of them, one request at a time, across four participants: the client, the application, the primary and a replica.

The shape to notice is that the write hands back a marker, and the next read presents that marker to decide where it is safe to read from.

sequenceDiagram
    participant C as Client
    participant A as App
    participant P as Primary
    participant R as Replica
    C->>A: POST /comment
    A->>P: INSERT
    P-->>A: ok, wal_lsn = 0/3A1F0C8
    A-->>C: 201 + X-Read-Token: 0/3A1F0C8
    C->>A: GET /post (X-Read-Token: 0/3A1F0C8)
    A->>R: replay_lsn?
    R-->>A: 0/3A1E900 (behind)
    A->>P: SELECT (fall back to primary)
    P-->>A: rows, including the new comment
    A-->>C: 200

Every label in that diagram is the mechanism. The client posts a comment, the app writes it to the primary and gets back a marker saying exactly where in the log that write landed, and it hands that marker to the client. On the next read the client presents the marker, the app asks a replica whether it has replayed that far, hears “no”, and reads from the primary instead.

An LSN is a log sequence number: a monotonically increasing byte offset into the write-ahead log. Postgres prints the 64-bit value as two hex halves separated by a slash, so 0/3A1F0C8 is the high word 0 and the low word 0x3A1F0C8 — byte 60,944,584 of the log. It is not a timestamp and it is not a row version; it is how far the log has been written, and because the log is append-only, comparing two LSNs is exactly asking which one happened first.

• P-->>A: ok, wal_lsn = 0/3A1F0C8 — the primary commits the INSERT and hands back the log position of that commit. This is a free byproduct of the write; you are not paying for an extra query.
• A-->>C: 201 + X-Read-Token: 0/3A1F0C8 — the app hands the position to the client rather than storing it, because the app tier is stateless (step 6) and the next request may land on any of 30 instances. A response header, a cookie, or a field in the JSON body all work; it only has to survive the round trip.
• C->>A: GET /post (X-Read-Token: 0/3A1F0C8) — the client echoes it back. The app now knows the exact freshness this particular user requires, which is the whole trick: there is no global staleness bound, only a per-session one.
• A->>R: replay_lsn? / R-->>A: 0/3A1E900 (behind) — the replica reports how far it has replayed, and replay_lsn < read_token means the answer is no. This single integer comparison is the entire “has this replica caught up” test.
• A->>P: SELECT (fall back to primary) — because the replica is behind, this read goes to the primary. The token does not make the read wait; it makes the read choose. With six replicas the app would try the others first and only fall back if none has caught up.

The gap in that comparison is worth converting into time, because it shows how sharp the test is:

commit position  0x3A1F0C8  =  60,944,584
replica position 0x3A1E900  =  60,942,592
gap                         =       1,992 bytes

log growth  =  694 writes/s x ~400 B  =  277,600 B/s  =  278 KB/s
time behind =  1,992 / 277,600        =  0.0072 s     =  7 ms

Seven milliseconds behind, well inside the 40 ms this replica normally runs at — and the token still says no. That exactness is the feature. A heuristic time window would have called this replica fresh and shipped the stale read.

The four fixes, priced

The table prices each fix at this chapter’s traffic, so you can compare them rather than pick by taste. max_expected_lag in the first row is a configured guess at the worst lag you expect a replica to have.

The three guarantee names, and the word they are defined against

Say these names out loud, because they are the vocabulary the interviewer is listening for.

The term all three are defined against is eventual consistency, and it is weaker than it sounds. It promises exactly one thing: if writes stop, every replica eventually converges on the same value.

It says nothing about when — 40 ms and four minutes both satisfy it — and nothing at all about what any single reader sees along the way. So it permits the trace above, permits a second read to return an older value than the first, and permits a reply to appear before the post it answers.

That is why eventual consistency gives you none of the three guarantees, and why each one has to be bought separately. An interviewer who hears “the replicas are eventually consistent” is waiting to hear which of the three you are adding on top.

Why fixed windows lose

Lag is bimodal, not smooth. It does not vary gently around an average; it lives in two separate regimes. It sits near zero, and it jumps to minutes during a bulk load or an index build (Replica lag is a correctness problem).

So a max_expected_lag of 500 ms is a bet you will lose during exactly the events that matter. Eject a replica from the read pool when its lag exceeds the window. It is the only fix that degrades instead of breaking.

Step 4 — Cache, and the ceiling that forces it

The replica fleet from step 3 works, but it pays for the same data seven times over — and that is what forces the next rung.

A cache is a small, very fast key-value store — almost always memory-resident, almost always Redis or Memcached — that holds the answers to recently asked questions under a key that identifies the question.

A read that finds its answer in the cache is a hit and costs half a millisecond. A read that does not is a miss and pays the full database cost behind it.

The measurement that forces a cache: seven copies of one hot set

Step 3 ended with six replicas and a spare at 10M DAU. The ceiling is RAM, and it is the only number in this chapter that gets worse the better the previous rung works.

The block below is in four parts: what the hot set weighs, what seven copies of it weigh, why one machine cannot hold it, and what the same reads cost with a cache instead.

a replica is a full copy, and must hold the hot set in RAM to keep the
97% buffer hit rate the 18 ms timeline query assumes

hot set at 10M DAU
  1e7 DAU x 2 posts/day x 400 B                 =  8 GB/day
  x 7-day hot window x 3 (rows + indexes)       =  168 GB

7 read copies (6 + spare)  =  7 x 168           =  1,176 GB of RAM,
                                                   holding the same
                                                   168 GB seven times

and the per-machine version is worse than the fleet version: at the
4 GB per vCPU of step 0's box (4 vCPU, 16 GB), an 8-vCPU replica has
32 GB, of which ~24 GB is page cache, so 168 GB is 14% resident and
the buffer hit rate falls well below the 97% the 18 ms timeline
query was derived at

the cache holds the hot 1% ONCE
  100,000 timelines x 20 posts x 400 B          =  0.8 GB
  and it removes 71% of the reads (derived below)
  6 + 1 replicas  ->  2 + 1
  RAM after       =  3 x 168 + 0.8              =  505 GB      ->  2.3x less

Replicas scale reads by duplicating the entire dataset. A cache scales reads by storing only the part that is actually read. At 10M DAU that is 1,176 GB against 505 GB, and every replica you add costs another 168 GB.

You can buy your way past the per-machine line, since memory-optimized instances exist and the six replicas of step 3 assume you bought them. But then you are paying seven times for one hot set, and the multiplier grows with traffic while the hot set does not. That is the forcing measurement, and it binds here, at 10M DAU.

Replicas do have a hard wall as well — each must replay 100% of the write stream — but that wall is at ~30M DAU and it is the argument of step 9. Do not spend one measurement on two rungs. An interviewer who hears the same number force two different decisions concludes you have one reason, not two.

Why the hot 1% is 71% of the reads

A cache’s hit rate can be derived from the shape of the read traffic, and doing so is the single most useful piece of arithmetic in the chapter, because it turns “add a cache” into a number.

Reads on a social feed are Zipf-distributed. That means popularity falls off in a specific way: rank the objects from most to least requested, and the probability of a request landing on the object at rank r is proportional to 1/r^s. A handful of objects take a large share of the traffic and the rest form a very long, very thin tail.

Take the exponent s = 1, the classic Zipf law and the value repeatedly measured for web page popularity, so p(r) is proportional to 1/r.

Probabilities have to add to one, so divide by their total. That total, over N objects, is the harmonic number H_N — the running sum 1 + 1/2 + 1/3 + ... + 1/N. The block below turns that into the formula worth memorizing.

p(r)  =  (1/r) / H_N          where H_N = 1 + 1/2 + 1/3 + ... + 1/N

hit rate of caching the top k  =  sum of p(1..k)  =  H_k / H_N

and since H_n  ~  ln(n) + 0.577 for large n, dropping the 0.577
from both halves gives the form worth memorizing

hit rate  ~  ln(k) / ln(N)

Dropping a constant from a numerator and a denominator is not free, so check it rather than trust it. Compute the ratio both ways at k = 1e5, N = 1e7:

exact      H_k / H_N     =  12.09 / 16.70  =  0.724
shortcut   ln(k) / ln(N) =  11.51 / 16.12  =  0.714
difference                                    1.4%

1.4% apart, which is far inside the error on “reads are Zipf-1” in the first place. Use the logs.

Substituting this chapter’s numbers

N is the number of distinct cache keys, not the number of rows in the database. Getting that wrong is the most common way this formula lies to you.

N = 10,000,000 distinct timelines -- one per DAU, and the timeline is
    what the 18 ms query builds and the cache replaces
  cache k =   100,000  (1% of the keys)   ->  11.51 / 16.12  =  0.714
  cache k = 1,000,000  (10%)              ->  13.82 / 16.12  =  0.857

Caching 1% of the keyspace removes 71% of the reads. The keyspace is the set of distinct keys the cache could ever be asked for, and it is what N counts.

Now the sensitivity, because it is the part a candidate gets caught on. Suppose you cache individual posts instead of whole timelines. The keyspace is no longer 10 million timelines; it is the 7-day post corpus:

posts in a 7-day window  =  1e7 DAU x 2 posts/day x 7 days  =  140,000,000
same k = 100,000         =  ln(1e5) / ln(1.4e8)
                         =  11.51 / 18.76                   =  0.61

61%, not 71% — which takes the replica count from 1.7 to 2.3, meaning from 2 machines to 3. Same formula, same k, one machine of difference, and the only thing that changed was what you decided an object was. State your N out loud and say what it counts.

Why you cannot just buy a 99% hit rate

Set the hit rate to 0.99 and solve for how many keys k that needs:

ln(k) / ln(1e7)  =  0.99
ln(k)            =  0.99 x 16.12  =  15.96
k                =  e^15.96       =  8.5e6
as a share of N  =  8.5e6 / 1e7   =  85% of the keyspace

Buying a 99% hit rate means holding 85% of the keyspace in RAM. The curve is logarithmic in k and therefore brutal at the top: the first 1% buys 71 points, and the last point of hit rate costs you the entire corpus.

That is why nobody buys 99%, and why “we’ll just raise the hit rate” is not a plan. The cost side of the same curve is tabulated in Estimation 5 a caches hit rate economics.

What the 71% actually buys

Two numbers. The first is the effective read latency, the average over hits and misses weighted by how often each happens. The second is how many replicas survive the cut, since only the misses still reach them.

effective read latency
  hit    0.714 x 0.5 ms                    =  0.36 ms
  miss   0.286 x (0.5 + 18) ms             =  5.29 ms
                                              -------
                                              5.65 ms   vs 18.0 ms uncached  ->  3.2x

replicas needed  =  6 x (1 - 0.714)  =  1.7   ->  2, plus 1 spare

The two patterns, since the recommendation is meaningless without them

There are exactly two ways to wire a cache to a database, and one of them is the wrong default here.

Cache-aside — also called lazy loading — means the application owns the cache, and the cache itself does not know the database exists. The application looks in the cache, and on a miss it queries the database and puts the answer back.

The pseudocode below shows both halves of that. The two lines to look at are in the write path, and the two paragraphs after the block explain why they are in that order.

READ  timeline(u):
    v = cache.get("tl:" + u)          # 0.5 ms
    if v is not None: return v        # 71% of the time, done
    v = db.query(TIMELINE_SQL, u)     # 18 ms, on a replica
    cache.set("tl:" + u, v, ttl=300)  # 5 min, so a missed invalidation self-heals
    return v

WRITE post(u, body):
    db.insert(POST_SQL, u, body)      # the database commits FIRST
    cache.delete("tl:" + u)           # then DELETE, never SET

Two terms in that listing need naming first.

A TTL, or time to live, is an expiry stamped on a cache entry, after which the cache forgets it whether or not anything has changed. ttl=300 means five minutes.

Invalidation is the act of removing a cache entry because the underlying data changed. A missed invalidation is one you forgot to do, which is why a TTL is worth having even when you invalidate correctly: it puts a five-minute ceiling on any mistake.

Now the two write-path lines.

db.insert comes before cache.delete so that the database is the only thing that has ever been told the truth. If the process dies between the two lines, the cache holds a stale entry that expires in 300 s. That is a staleness bug with a known bound, not a lost write.

delete rather than set means you never write a value into the cache that no reader asked for and no transaction ordered. Two concurrent writers doing set can leave the older value resident forever. Two concurrent deletes cannot leave anything wrong at all.

Read-through, and why it is not the default

Read-through inverts the ownership: the application only ever calls the cache, and the cache library fetches from the database on a miss.

It is less code, and it is the wrong default here. The cache is now on the write path’s correctness boundary — a read-through cache that is up but wrong is indistinguishable from a database that is wrong, and there is no layer left to fall back to.

With cache-aside, the failure mode of the cache is a latency spike. With read-through it can be a correctness bug. That asymmetry is the entire recommendation.

Three more decisions that carry numbers

• Size for the miss path, not the hit path. At a 71% hit rate you run 2 replicas; a cold cache — one that has just been emptied and knows nothing — instantly demands 6. Either keep 6 (and save nothing) or accept that a cache flush is an outage and warm the cache before taking traffic. This is the single most common cache incident.
• The eviction policy is a working-set question. Eviction is how a full cache decides what to throw out to make room. LRU — least recently used, discard whatever has gone longest untouched — is right when the hot set is stable; for a feed where yesterday’s post is dead, a TTL plus LRU beats either alone, because the TTL removes what is old and LRU removes what is unpopular.
• Invalidation races and versioned keys are derived in Caching layers and invalidation. Cite them and move on — re-deriving them burns four minutes you need for sharding.

The thundering herd, which the TTL itself creates

The thundering herd — also called a cache stampede — is what happens when a popular key expires and every request for it misses at the same instant, so many callers all run the same expensive query to produce the same answer.

Its cost comes from Little’s law: the number of things in flight equals the arrival rate times how long each one lasts.

The block below applies that to two different herds. The first is one very popular key expiring. The second is a hundred thousand keys expiring together because they were all warmed at the same moment.

one hot key: 1/H_N = 1/16.7 = 6.0% of reads  =  0.060 x 6,250  =  375 /s
  in flight during one 18 ms rebuild         =  375 x 0.018    =  6.7 duplicate queries

mass expiry: the top 100,000 keys were warmed together and share
one TTL, so they expire together
  reads onto the miss path                   =  0.71 x 6,250   =  4,438 /s
  in flight at 18 ms each                    =  4,438 x 0.018  =  80 concurrent queries
  against a pool whose optimum is                                34

Those are two different failures and they need two different guards, which is why the incident recurs at teams that only deployed one.

• Single-flight fixes the first. Per process, the first caller to miss a key takes a lock on that key, runs the query, and every other caller for the same key waits on that one result. The 6.7 duplicate queries become 1. It does nothing at all for the second case, because those are 100,000 different keys and single-flight already allows one query each.
• TTL jitter fixes the second. Write ttl = 300 + random(0, 60) so the same warm-up does not become a synchronized alarm clock, spreading those 4,438 misses over a minute instead of an instant.
• Probabilistic early expiry covers both at once: refresh a key at random, with the probability rising as its TTL approaches zero. The cost is a few percent of extra queries all the time.

Step 5 — CDN, forced by bytes

A CDN is a fleet of servers, rented from a provider, spread across hundreds of cities. Each one is called an edge, and each holds copies of your static files — images, JavaScript, stylesheets — so that a user fetches them from a machine ten milliseconds away instead of from your own origin, meaning the servers you actually run.

Nothing above this rung was ever about bandwidth. This one is entirely about it — the first rung forced by raw bytes rather than by processor time.

Argument one: the machine count

The block converts a day’s worth of static bytes into a number of machines. Follow the units — bytes per day, then bytes per second, then bits per second, then machines:

10M DAU x 5 page loads/day x 2 MB of assets and images  =  100 TB/day
average egress   =  100e12 B / 86,400 s  =  1.16 GB/s
in bits          =  1.16 x 8             =  9.3 Gbps
peak 3x                                  =  27.8 Gbps
at 1 Gbps/machine, 50% usable            =  27.8 / 0.5  =  55.6  ->  56 machines

Serving static bytes needs 56 machines. The entire application needs 30. That comparison is the argument, and it is a machine count rather than an opinion.

Volunteer the assumption that count rests on, because it is the one an interviewer will push, and because this chapter deliberately does not take the default the rest of this track uses.

1 Gbps at 50% usable is the pessimistic end of the commodity range for a machine’s network throughput. Numbers worth memorizing cold says to default to 10 Gbps and to say so out loud, and on 10 Gbps the same 27.8 Gbps needs 27.8 / 5 = 5.6 → 6 machines — a fifth of the app tier rather than nearly double it.

Both counts are defensible; silently picking one is not. The defence for the low end here is that origin egress at this shape is rarely limited by the network card itself. It is limited by TLS termination — the processor cost of setting up each encrypted connection — and by connection state on the same boxes that serve the 5% of requests the CDN misses. So the effective per-box figure sits near the bottom of the range even when the card does not.

The conclusion survives either way, because the real argument is that 100 TB/day is a bandwidth business you do not want to be in, not that the number is 56.

Argument two: latency

Do this one for a user sitting 75 ms away from the origin.

TTFB is time to first byte: how long after the click the first byte of the response arrives. RTT is round-trip time, one message out and its answer back.

Three round trips have to complete before that first byte can arrive: one to establish the TCP connection, one for the TLS handshake (transport layer security, the encryption every HTTPS page uses), and one to send the request and get the answer.

origin, 75 ms RTT
  TCP handshake      1 RTT   =   75 ms
  TLS 1.3            1 RTT   =   75 ms
  request + response 1 RTT   =   75 ms
                                ------
  time to first byte         =  225 ms

edge, 10 ms RTT              =  3 x 10  =  30 ms      ->  7.5x faster

Argument three: the bill, which is the largest of the three

Compare serving all the bytes yourself against serving 95% of them from the edge. Origin egress is $0.05 per gigabyte, CDN egress $0.02. Offload is the fraction of bytes the edge serves without ever asking your origin.

The monthly volume comes from the 100 TB/day above: 100,000 GB/day x 30 days = 3,000,000 GB/month.

no CDN   3,000,000 GB/month x 0.05                      =  $150,000/month
CDN at 95% offload
  edge     2,850,000 GB x 0.02                          =  $ 57,000
  origin     150,000 GB x 0.05                          =  $  7,500
                                                            --------
                                                            $ 64,500/month

$150,000 down to $64,500 is 2.3x, or $85,500 a month saved.

Price the machines too, so the comparison is a number rather than a gesture:

56 four-vCPU boxes  =  56 x 4 x $0.05/vCPU-hour x 730 hours  =  $8,176/month
against the egress saving                                    =  $85,500/month

The bandwidth is the argument; the machines are a rounding error on it.

Three things to say about CDNs

• Cache key discipline is where CDNs actually fail. The edge decides whether two requests are “the same” by building a key out of the URL plus whatever the Vary response header tells it also matters. A Vary: Cookie on a static asset says every distinct cookie is a distinct object, so each user gets their own copy, the hit rate drops to near zero and the bill goes back to $150,000. Serve static files from a path that never sets cookies.
• A long TTL plus content-hashed filenames removes invalidation entirely. Put a hash of the file’s contents in its name — app.4f1c8e.js — and give it a one-year expiry: it can never be stale, because changing the contents changes the name, and it never has to be purged from hundreds of edges.
• The CDN is also your first absorber of a DDoS, a distributed denial-of-service attack in which many machines flood you with requests at once. The edge fleet has far more capacity than you do and soaks the flood before it reaches your origin, and that is often the larger reason to have one.

Step 6 — Make the web tier stateless

Like the CDN, this rung is not forced by processor time. It is forced by the error budget, and it turns on one distinction.

A server is stateful when it remembers something between requests that no other server knows — typically a user’s logged-in session, held in its own memory. It is stateless when it remembers nothing, so any server can serve any request from any user.

A stateful web tier forces sticky routing: the load balancer must send each user back to the same instance every time, because only that instance holds their session.

The error-budget calculation that kills sticky routing

Sticky routing is fine until it is priced against a service level objective.

An error budget is the way to price it. If you promise 99.9% availability, you are promising that no more than 0.1% of requests fail — and that 0.1% is a budget you get to spend.

Take 10M DAU with 30 instances and a 99.9% target. The block computes the budget, then the cost of replacing one instance, then what a month of deploys spends.

This is the one place the chapter deliberately mixes the two rates, so it says so: the budget is a volume, so it comes from the average; a replacement window is a worst case, so it comes from the peak.

monthly requests  =  2,315 avg QPS x 86,400 x 30    =  6.0e9
error budget      =  6.0e9 x 0.001                  =  6,000,000 requests

one instance replacement, sticky sessions
  1/30 of traffic fails for 60 s while clients re-auth
  = 6,944 peak QPS / 30 x 60                        =  13,888 requests
  =  0.23% of the monthly budget

deploys: 30 instances x 2 deploys/week x 4.3 weeks  =  258 replacements/month
  258 x 13,888                                      =  3,583,000 requests
  =  60% of the entire monthly error budget, spent on deploys

Sticky sessions do not fail the throughput test. They fail the error-budget test.

Autoscaling — automatically adding and removing instances as load changes — makes it worse, because scaling from 30 instances down to 10 overnight evicts two-thirds of live sessions on purpose.

Three places to put the session instead

The table below compares them. Two terms first.

Revocation is the ability to cancel a login immediately — on a logout, a password change, or a stolen laptop. It is the property the middle option cannot offer.

A JWT is a JSON web token: a small blob of user facts that the server signs cryptographically, so any server can verify it is genuine without looking anything up.

Why the denylist stays tiny

The third option needs a revocation list, sometimes called a denylist: the set of tokens that have been cancelled before their natural expiry and must be refused.

The objection is always that it will grow without bound. The arithmetic ends that argument, because an entry only has to be kept until the token it names would have expired anyway — this is Little’s law again, with revocations as the arrival rate and the token lifetime as the time each entry stays:

1,000 revocations/day, 15-minute (900 s) token lifetime
  arrival rate  =  1,000 / 86,400        =  0.0116 revocations/s
  live entries  =  0.0116 x 900          =  10.4 entries

Ten entries. A 15-minute access token with a denylist gives you both zero-lookup validation in the common case and instant revocation, and the denylist fits in a variable.

One authenticated request, traced

“Stateless” is doing a lot of work as a word, so trace one request through the third option. The mechanism is four steps, and the thing to notice is that steps 1 to 3 touch no network at all.

Three fields of the token matter. sub is the subject, the user the token is about. jti is the token id, a unique name for this particular token so it can be revoked individually. exp is the expiry, as a count of seconds.

HMAC-SHA256 is a keyed signature: a hash of the payload mixed with a secret key that only your servers hold. Anybody holding the key can check it, and nobody without the key can forge it.

the cookie holds one signed token, ~400 B
  {"sub": "u_8814", "jti": "t_4f1c", "exp": 1735689600, "scope": "user"}
  + HMAC-SHA256(header + payload, server_key)

1. app recomputes the HMAC over the payload with the shared key   ~2 us CPU, 0 I/O
   -> the token is genuine, and NOTHING was looked up
2. app compares exp against the clock                             expired -> 401
3. app checks jti against the in-process denylist                 dict lookup on
   (a set of ~10 strings, refreshed from Redis every 1 s)         ~10 entries
4. request proceeds with sub = u_8814

Step 1 is why this scales. The signature proves your servers minted the token, so establishing identity costs no network round trip and any of the 30 instances can serve any request.

Step 3 is why it is still revocable. A logout writes t_4f1c into a tiny shared set, every instance picks it up within one second, and 15 minutes later the entry is dropped because the token would have expired anyway.

At minute 15 the token simply stops verifying at step 2. The client then presents its long-lived refresh token to get a new one, and that refresh — once per user per 15 minutes, not once per request — is the only part of authentication that touches a database.

The rest of the state has to leave too

• Uploaded files go to object storage, a service that stores whole files under a name and is reachable from every instance, such as Amazon S3. Never to a local disk: a scale-in, the automatic removal of an instance, would delete user data with it.
• Background job state goes to the queue introduced in step 8.
• Rate-limit counters go to the shared store. If each instance counts separately, thirty instances each allowing 100 requests is an effective limit of 3,000.

Step 7 — Multiple data centers

Running the same stack in two parts of the world sounds like a traffic problem. The hard part is the data.

A region is a cloud provider’s cluster of datacenters in one geography — Northern Virginia, Frankfurt. Machines inside one region are milliseconds apart. Machines in different regions are tens of milliseconds apart.

Two independent things force a second region, and they must not be conflated.

Forcing function one: latency

30% of 10M DAU sit 75 ms away from the single region, using the round-trip constant established in Scaling replication.

Loading a page costs six serial round trips, not one: three to get the first byte of the page (as step 5 derived) and three more for the API calls the page then makes. Serial means each one has to finish before the next can start, so they add rather than overlap.

time to first byte                          =  3 x 75  =  225 ms
3 serial API calls the page needs           =  3 x 75  =  225 ms
                                                          ------
                                                          450 ms before render

same page in an in-region deployment, 5 ms RTT
  3 x 5 + 3 x 5                                        =   30 ms   ->  15x

Forcing function two: availability

A whole region is itself a correlated failure domain. One power event, one bug in the provider’s control plane — the software that manages the machines, as opposed to the machines themselves — or one bad configuration pushed region-wide takes all of it at once.

Two regions do not multiply out to seven nines, for the same reason two instances did not. But they do remove the single largest correlated failure domain you have left.

Where the data lives

The hard part of running two regions is not steering the traffic, which is a DNS setting. It is deciding where the data lives.

The diagram shows the arrangement this chapter recommends. Read it for one asymmetry: solid arrows are the request path, dashed arrows are the things that happen without anyone waiting, and only one box in the picture accepts writes.

flowchart TD
    U1(["EU user"]) -->|"geoDNS · 5 ms"| E["EU region<br/>app + cache + replicas"]
    U2(["US user"]) -->|"geoDNS · 5 ms"| W["US region<br/>app + cache + PRIMARY"]
    E -->|"reads: local replica"| ER[("EU replica")]
    E -->|"writes cross · 75 ms"| P[("US primary")]
    W -->|"reads and writes stay local"| P
    P -.->|"async WAL · lag 75-200 ms"| ER
    ER -.->|"failover promotes this replica<br/>bounded data loss = lag"| E

    style P fill:#1d3557,color:#fff
    style ER fill:#2d6a4f,color:#fff

The colours are the chapter’s key: the blue box is the authoritative primary, the green one is a replica that takes read load off it. The two region boxes are left uncoloured because a region is a place, not a component.

geoDNS — DNS that answers with a different address depending on where the query came from — puts each user 5 ms from an application tier, so both region boxes are within a local round trip of their users.

The two regions are not the same stack. The EU region has an app tier, a cache and replicas, but no writable database. The US region has all of that plus the primary, which is the authority.

That produces the asymmetry:

• In the US, reads and writes stay local.
• In the EU, reads go to the local replica and cost 5 ms.
• In the EU, writes cross the ocean to the primary and cost 75 ms.

Since the workload is 90% reads, that asymmetry is the whole design. The average penalty for an EU user is 0.1 x 75 ms = 7.5 ms, which is the first row of the table below.

The dashed arrows are the two things nobody waits for. Async WAL shipping carries committed log records from the primary to the EU replica; because the primary does not wait for an acknowledgement, the replica is behind by the network round trip plus whatever it is queueing behind, which is 75–200 ms in normal operation.

That range is why the failover arrow is dashed too. Promoting the EU replica when the US region is gone is a decision you can make in seconds, but it publishes a database that is missing everything in that gap.

Four data strategies, priced

Two terms in the table need naming.

OLTP is online transaction processing: the ordinary read-write traffic of an application, as opposed to analytics. It is the workload where a 75 ms wait per write is fatal.

A CRDT is a conflict-free replicated data type, a data structure designed so that two copies edited independently can always be merged without a human choosing a winner. The alternative is last-writer-wins, which resolves conflicts by discarding one of the two edits.

How much a failover loses

This is the number to volunteer. Take the middle of the diagram’s 75–200 ms range: the low end is the physical floor and the high end is a bad minute, so 150 ms is the number to quote and the range is the number to defend.

With asynchronous replication and 150 ms of lag, a hard regional failover loses up to 150 ms of committed writes:

writes in flight at failover  =  694 writes/s x 0.15 s  =  104 transactions

Whether 104 lost posts is acceptable is a product question. Whether 104 lost payments is acceptable is not.

Step 8 — Message queue, and the concurrency arithmetic

This rung is orthogonal to the last two. Nothing in step 7 causes this one. The arithmetic below holds at 1M DAU exactly as it does at 10M, and if your product has one slow endpoint you should do this immediately after step 2. It sits here because it is easiest to see once the fleet has a size.

The measurement that forces slow work off the request path is about concurrency, not throughput.

A message queue is a durable list. Instead of doing a slow job while the user waits, the web server appends a short description of the job to the list and answers immediately. Separate machines called workers pull jobs off the list and do them.

A thread is one unit of a server’s ability to have a request in progress, and a server has a fixed pool of them.

The forcing measurement is Little’s law

Little’s law says that for any stable system, L = lambda x W. The average number of items inside the system equals the arrival rate times the average time each one spends there.

It needs no assumption about the arrival distribution or the service distribution — it is an accounting identity — which makes it the safest formula in this chapter.

Map the three symbols onto this problem: L is threads held, lambda is requests per second, and W is how long a request holds a thread.

Now the numbers. A photo post resizes into 5 variants at 200 ms of processor time each, so its W is a full second, while every other request’s W is the 8 ms from the workload block.

photo posts  =  10% of writes  =  694 x 0.1  =  69 /s
work per post  =  5 x 200 ms                 =  1.0 s

L = lambda x W, photo work        =  69 x 1.0        =  69 threads
L = lambda x W, everything else   =  6,944 x 0.008   =  56 threads

Both lines are the same multiplication, and that is the point.

A thread is held for the duration of the work, so a request that is 125x slower costs 125x the concurrency at equal rate — 1.0 s / 0.008 s = 125. That is the entire reason 1% of the traffic outweighs the other 99%.

1% of requests need more concurrency than the other 99% combined. And the failure is not slowness. It is head-of-line blocking: the photo requests occupy the thread pool, and the 8 ms requests queue behind them the way one slow shopper holds up a whole checkout line. The p99 of unrelated endpoints collapses.

What the queue buys, part one: the thread comes back

With a queue in place the web tier only performs an enqueue — appending the job description to the list — and the thread is released immediately. Same formula, new W:

enqueue cost                     =  1 ms
concurrency held                 =  69 x 0.001  =  0.07 threads

69 threads become 0.07.

What the queue buys, part two: burst absorption

This is what turns a spike into a delay, so it is worth quantifying. The block asks what happens if photo posts arrive at ten times their peak rate for a full minute.

offered load, photo posts at peak      =  69 /s
worker CAPACITY, provisioned at 2x peak = 138 jobs/s   (100% headroom)
a 10x burst for 60 s
  arrivals                             =  690 x 60  =  41,400
  served during the burst              =  138 x 60  =   8,280
  backlog at t = 60 s                  =  33,120
  drain rate = capacity - offered load =  138 - 69  =  69 /s
  drain time                           =  33,120 / 69  =  480 s  =  8 minutes

Drain time is set by the headroom, not by the throughput. The 138 - 69 subtraction is the entire recovery story: a worker fleet provisioned at exactly 69/s has a surplus of zero and never drains at all. Consumer lag backpressure and the rebalance storm prices the same subtraction at 20% and 5% headroom.

A 10x burst becomes an 8-minute delay in thumbnail availability instead of a site outage. That is the trade, and stating it that way is the point.

Three properties of a queue worth naming

Each one is a thing that goes wrong if you skip it.

• Consumer lag is the metric to watch, not throughput. Consumer lag is how far behind the workers are — the size of the backlog, or better, how long it would take to clear. Throughput looks perfectly healthy right up to the moment the backlog is growing without bound, because a saturated worker fleet is running flat out. Alert on backlog / drain_rate, which comes out in seconds and is directly interpretable.
• Handlers must be idempotent, because delivery is at-least-once. At-least-once delivery means the queue guarantees a job is delivered but not that it is delivered only once — a worker that crashes after finishing but before acknowledging will see the job again. Idempotent means running an operation twice leaves the same result as running it once. Key every job on a deterministic id, so a duplicate resize overwrites the existing thumbnail rather than appending a second one.
• Run a dead-letter queue with a size alarm. A dead-letter queue is where a job goes after it has failed too many times, so it stops being retried forever. Without one, a single poison message that fails and retries endlessly is indistinguishable from a healthy queue on a throughput dashboard.

Step 9 — Sharding, last

The last rung is the one you cannot undo, which is why it arrives with three forcing measurements, six cheaper alternatives, and a way of doing it that does not trap you.

To shard is to split the data itself across several databases, each holding a disjoint slice of the rows. Which database a row lives on is decided by the value of its shard key — one chosen column.

Sharding is the only move in this chapter that adds write capacity, and the only one you cannot cheaply undo.

The three measurements that force it

None of them is “the database is slow.” Each block below is a separate ceiling, and any one of them is enough.

1. REPLICA APPLY
   every replica replays 100% of the write stream, and apply is far
   less parallel than the primary's write path
   single-threaded logical apply, per write =  2 ms
   single-threaded ceiling  =  1 / 0.002    =    500 writes/s
   parallel apply (~4 streams, relaxed ordering)
                            =  4 x 500      =  2,000 writes/s
   the traffic ladder crosses that at 30M DAU:
     30e6 DAU x 2 writes/day / 86,400 s x 3 =  2,083 writes/s
   -> adding replicas buys read capacity and exactly zero write capacity,
      and past this point the replicas cannot keep up with the primary
      at all, no matter how many you run

2. WORKING SET
   posts at 100M DAU  =  1e8 x 2 x 400 B   =  80 GB/day
   7-day hot window x 3 (rows + indexes)   =  80 x 7 x 3  =  1,680 GB
   -> needs a 2 TB-RAM instance, which is the top of the SKU list

3. RESTORE TIME
   one year of posts                       =  80 x 365     =  29,200 GB
   restore at 1 GB/s sequential            =  29,200 s     =  8.1 hours
   -> that is your recovery-time objective (RTO) floor -- the fastest you
      could be back after losing the data -- and adding replicas does not
      move it, because a logical corruption replicates too
   sharded 16 ways                         =  29,200 / 16  =  1,825 s = 30 min

The restore-time argument is the one nobody makes and the one operators find most convincing.

A single 29 TB database has an 8-hour recovery-time objective no matter how many replicas you run. The reason is logical corruption — a bad migration or a buggy UPDATE that writes wrong values rather than damaging the disk. That is a legitimate write, so every replica faithfully applies it.

Replicas protect you from a machine dying. They do not protect you from a mistake.

Do these six things first, in this order

The table lists the cheaper moves that come before sharding, in the order to try them. Six of the seven are reversible and the seventh is not, which is the whole reason for the ordering.

Sharding is the only item on that list you cannot undo cheaply, which is why it is last. Shard key selection, hot shards, hash vs range, the mod N 94% reshuffle, consistent hashing, and the cross-shard cost table are all derived in Scaling partitioning sharding pooling caching — cite it rather than re-derive it.

The operational shape: three decisions

What this chapter adds on top of that reference is how to shard so the decision does not trap you.

• Fix a large number of logical shards up front. A logical shard is a bucket that a row is assigned to permanently. A physical node is a machine, and many logical shards live on one. Route on hash(user_id) mod 1024 — never mod 16 — and keep a lookup table from logical shard to physical node:

  logical shards	physical node
  0 – 63	node 0
  64 – 127	node 1
  …	…
  960 – 1023	node 15

  Adding node 16 means editing rows in that table and copying the 64 logical shards you reassigned. About 6% of the data moves — 64 / 1024 — and every key’s logical shard number is unchanged, so no row is ever rehashed and no client needs to know a node count. Route on mod 16 directly and adding a node changes the destination of ~94% of keys (Scaling partitioning sharding pooling caching); that retrofit is the quarter-long project.

• Shard by user_id, not by post_id. The dominant query is “this user’s timeline”. Sharding by post makes every timeline read a scatter-gather — one query fanned out to all 16 shards, with the answers merged — and a scatter-gather is only as fast as its slowest participant. Your p99 becomes the maximum over 16 shards instead of the value for one. Put a number on it: for the whole to be fast, all 16 have to be fast, so if p is the per-shard chance of being fast then 0.99 = p^16, giving p = 0.99937. A 99th-percentile promise on the fan-out is a 99.94th-percentile demand on each shard.

• The migration is the project, not the schema. The sequence is five steps, and each has a window in which correctness depends on the previous step being finished:

  1. Dual-write every change to both the old and the new layout, so they stay in step.
  2. Backfill the historical rows into the new layout, throttled over days so it does not swamp the primary.
  3. Verify with range checksums — hash each contiguous block of rows on both sides and compare — so you learn that the copies agree rather than assuming it.
  4. Flip reads over one slice of traffic at a time.
  5. Only then stop dual-writing.

  Step 3 is the one teams skip and regret.

The architecture, assembled

All ten rungs now fit on one page, every box carrying the number that forced it — and a single request can be walked through the whole thing.

If you remember one diagram from this chapter, it is this one. The point of it is that you can recite the derivation of every label on it. Read it top to bottom as the path a request takes, and note that the dashed arrows are the things nobody waits for.

flowchart TD
    U(["Users"]) --> DNS["geoDNS"]
    DNS --> CDN["CDN edge<br/>95% offload · 30 ms TTFB"]
    CDN -->|"5% miss"| LB["Load balancer<br/>least-outstanding-requests"]
    LB --> W["Web tier · 30 x 4 vCPU<br/>stateless · rho 0.7"]
    W --> SS[("Session + rate limits<br/>Redis")]
    W --> C[("Cache · 1% of corpus<br/>71% hit rate")]
    W --> POOL["Connection pooler"]
    W --> Q[["Queue"]]
    Q --> WK["Workers<br/>resize · fan-out · email"]
    POOL --> PRI[("Primary<br/>writes + read-your-writes")]
    POOL --> REP[("Replicas x 2<br/>warm spare x 1")]
    PRI -.->|"async WAL"| REP
    WK --> PRI
    W --> OBJ[("Object storage")]
    CDN -.->|"origin pull"| OBJ
    ALL["Metrics · logs · traces<br/>sampled at 2.1%"] -.- W

    style CDN fill:#bc6c25,color:#fff
    style C fill:#2d6a4f,color:#fff
    style REP fill:#2d6a4f,color:#fff
    style PRI fill:#1d3557,color:#fff
    style Q fill:#40916c,color:#fff
    style ALL fill:#495057,color:#fff

Same colour rule as the ladder: blue is authoritative state, green is read capacity (the cache and the replicas), light green is work taken off the request path without answering a read (the queue), orange is a box forced by something other than processor time, and grey is the thing that watches.

Now walk one timeline request through it. Every bullet is one box, and every box gets its number back.

• geoDNS answers with the nearest region, so the first packet travels 5 ms rather than 75 (step 7).
• The CDN edge serves the page shell, the JavaScript and the images — 95% offload, 30 ms TTFB against 225 ms from the origin (step 5). What it does not have, it fetches by origin pull from object storage; that is the dashed arrow, and it happens once per asset per edge per TTL, which is why content-hashed filenames and a one-year TTL make it effectively never.
• The remaining 5% miss is the only traffic that reaches the load balancer, which picks the instance with the fewest outstanding requests rather than the next one in the ring (step 2).
• The web tier is 30 four-vCPU instances at rho 0.7 — 20 for the load, 30 so that losing an availability zone is uneventful (step 2). It is stateless, so the request carries its own identity: a signed token verified locally, checked against the ~10-entry denylist in the session and rate limits store, which is the same Redis (step 6). Rate limits live there and not in process memory, because 30 instances each counting to 100 is a limit of 3,000.
• Reads try the cache first — 1% of the corpus resident, a 71% hit rate, 0.5 ms on a hit against 18 ms on a miss (step 4).
• Everything that misses goes through the connection pooler, without which 30 instances x 20 connections is 600 against an optimum of 34 (step 2). Behind it, the primary takes writes plus the read-your-writes reads that must not be stale, and replicas x 2 plus a warm spare x 1 take everything else. The primary ships async WAL to them, and that shipping is the lag every consistency fix in step 3 is fighting.
• Anything holding a thread for longer than a few milliseconds goes on the queue instead, where workers resize images, fan out writes and send email off the request path (step 8). Note that the workers write to the primary, not to a replica: they are producers of state, not readers of it.
• Finally the grey box holds the observability plane: metrics, which are pre-aggregated counters and timings; logs, which are one text record per event; and traces, which stitch together every hop a single request made across every service. The grey box is labelled metrics · logs · traces sampled at 2.1%, because only the traces are sampled, and that sampling keeps every error and every slow request whole. The next section is about why that percentage is the interesting number.

Observability, and the bill that is not where you think it is

The logging pipeline has a bill too, and the input that actually moves it is not the one everybody estimates.

Start with the estimate everyone writes, and carry the units at every step:

10M DAU, 200e6 requests/day, 1 KB of structured log per request
  volume                       =  200e6 x 1,000 B   =  2e11 B  =  200 GB/day
  at $0.10/GB ingested         =  200 x 0.10                   =  $20/day
  monthly                      =  20 x 30                      =  $600/month

$600/month against an app tier of $4,380/month. Logging is 0.14x the compute it observes, not a multiple of it.

That conclusion is fragile in exactly one way, so guard it. Mistake megabytes for gigabytes anywhere in that chain and the same inputs produce 200,000 GB/day — a bill 137x the compute, and the opposite architectural conclusion. Carrying the raw byte count 2e11 B before converting to 200 GB is what makes the slip impossible.

One kilobyte per request is not expensive. Say the true number first, then go and find the real one.

Solving for the input that does move the bill

Invert the arithmetic. Instead of asking what logging costs, ask how many bytes per request it would take for logging to cost exactly what compute costs:

app tier per day             =  4,380 / 30                     =  $146/day
GB/day at $0.10/GB           =  146 / 0.10                      =  1,460 GB/day
bytes per request            =  1,460e9 / 200e6                 =  7,300 B  =  7.3 KB

7.3 KB per request is the crossover, and that is roughly thirty structured log lines. That is not a pathological number; it is a normal request traversing a normal call graph.

So the 1 KB assumption was never the realistic one. A request does not emit one log line — every service it touches emits its own, and the count multiplies:

20 services x 1 KB per service per request                      =  20 KB/request
volume                       =  200e6 x 20,000 B  =  4e12 B     =  4,000 GB/day
bill                         =  4,000 x 0.10                    =  $400/day
monthly                      =  400 x 30                        =  $12,000/month
vs the app tier              =  12,000 / 4,380                  =  2.7x

The lesson is not “logging is expensive.” It is that a logging estimate has no meaning until you state bytes per request summed over the call graph, and that the fan-out — not the line size — is what crosses the compute bill. The 20-service graph rejected in alternatives for its latency turns out to be the logging bill too.

The fix, and why 2.1% loses nothing

Head-based sampling means deciding at the moment a request starts whether to record it. Keep everything that went wrong, and keep only a small random slice of the rest.

The block sums the three categories you keep, then divides by the day’s requests to get the sampling rate that falls out.

keep 100% of errors        =  200e6 x 0.001         =    200,000 /day
keep 100% of the slow tail =  200e6 x 0.01          =  2,000,000 /day
1% sample of the remainder =  197.8e6 x 0.01        =  1,978,000 /day
                                                       ---------
                                                       4,178,000 /day  =  2.1%
bill at 20 KB/request      =  0.021 x 400           =  $8.40/day  =  $252/month
vs the app tier            =  252 / 4,380           =  0.06x

Head-based sampling that keeps every error and every p99 request costs 2.1% of the bill and loses nothing you would have looked at. It takes observability from 2.7x the compute to 0.06x of it.

Metrics and traces are different products: metrics are pre-aggregated, cheap and always on, while traces are sampled, expensive and for the tail. Conflating them is how the $12,000 line appears in the first place.

Failure modes

Knowing the rung is only half of knowing the design; the other half is the way each rung fails in production.

Every row is one rung failing. The four columns are what breaks, the number that proves it broke, the signal that would have told you, and the fix. An SLI in that third column is a service level indicator: a measurement you have chosen to watch as a proxy for health.

Read the detection column first. Most of these are undetectable on the dashboard you would naturally build. A queue with an unbounded backlog has perfect throughput. A thundering herd is a miss-rate spike with no traffic change. A connection-starved tier shows healthy CPU. The guards in the last column are cheap; noticing is the expensive part.

Alternatives rejected

A reasonable person would propose several other designs, and each has something genuinely right about it. Being able to argue that other side — and then name the measurement that rules it out at this scale — is most of what separates a designed answer from a memorized one.

Interviewer pushback

Eight questions are how this material actually gets tested, so each appears below with the answer written the way it should be spoken. The italic line under each question names what it is really probing.

“You have 10 million users. Where do you start?” Testing: whether you size before you draw. With the numbers, not the diagram. 10M DAU at 20 requests each is 200 million requests a day, which is 2,315 average QPS and about 6,900 at a 3x peak. At 8 ms of application CPU that is 56 cores of work, so 80 cores at a 70% target, so 30 four-vCPU instances once I want to survive losing one of three availability zones. The database is 5 ms per request of which 4.5 is reads, so the read path leaves the primary and the write path is 694 per second, which one primary handles comfortably. Then I draw, and every box on the drawing has a number under it.

“Why not just buy a bigger machine?” Testing: whether “horizontal” is a reflex or a conclusion. You should, first — going from 4 to 192 vCPU is 48x on this workload and it is one line of configuration, so it covers us from 300,000 DAU to about 15 million. What it does not cover is availability: one box is one failure domain, and 57% of its 3.1 hours of annual downtime is deploys, which simply vanish with two boxes. So the honest answer is that vertical scaling solves the throughput problem for years and never solves the availability problem for a day, and availability is what forces the second machine.

“Add a cache. How much does it help?” Testing: whether a cache is a shape or a number. Depends entirely on the access distribution, so I would measure it. For Zipf with exponent one, the mass under the top k of N is the harmonic ratio H_k/H_N, which is ln(k)/ln(N). My N is ten million distinct timeline keys — one per daily active user — so caching 100,000 of them, one percent, gives 71%. I would say the N out loud, because if I cached individual posts instead the keyspace is 140 million and the same 100,000 keys give 61%, which is a whole extra replica. That takes effective read latency from 18 ms to 5.65 ms and replica count from 6 to 2. The number I would actually worry about is the miss path: at 71% hit I need 2 replicas, and the instant that cache is empty I need 6. So either I keep 6 and save nothing, or I accept that a cache flush is an outage and I build warming into the deploy. Most cache incidents are that, not eviction policy.

“A user posts a comment and doesn’t see it. What happened and what do you do?” Testing: read-your-writes, and whether you know it is a correctness bug. They wrote to the primary, got redirected, and the read landed on a replica that was 40 ms behind. It looks like staleness but it is a lost-update bug, because the user re-submits a form rendered from the stale value and overwrites the good row. Three fixes, and I would ship the first. A sticky-primary window routes that session’s reads to the primary after a write — it costs one primary read per write, and comparing the two at peak, 694 writes against 6,250 reads, that is 11% of read traffic, not 100%. LSN tokens are exact: return the write’s log position and only serve from a replica that has replayed past it. And separately I would pin each session to one replica for monotonic reads, so time never runs backwards even outside the write window. The trap is that lag is bimodal — it sits at 40 ms and jumps to minutes during an index build — so any fixed window is a bet, and I would eject lagging replicas from the pool as well.

“When do you shard?” Testing: whether sharding is a last resort or a first instinct. Three measurements, and I would want all three before doing it. Replica apply is the surprising one: each replica replays the full write stream, and parallel apply tops out near 2,000 writes per second, which we hit around 30 million DAU — so past that, replicas cannot keep up regardless of how many I add. Second, the working set: at 100 million DAU a seven-day hot window with indexes is 1.7 TB, which is the top of the instance list. Third, and the one people forget, restore time: a year of posts is 29 TB, which restores at 1 GB/s in 8.1 hours, and that is my RTO floor no matter how many replicas exist, because logical corruption replicates. Sixteen shards makes it thirty minutes. Before any of that I would index, cache, add replicas, scale vertically, split one workload out, and archive cold rows — sharding is the only one of those I cannot undo.

“Your p99 is 400 ms and CPU is at 80%. What is going on?” Testing: whether you know queueing. 80% utilization is the problem, not a symptom of one. Response time scales as 1/(1-rho), so at 80% the queue multiplier is 5 and the p99 is roughly 4.6 x service_time / 0.2, which is 23 service times. If service time is 17 ms, that is your 400 ms exactly, and no code is slow. Two things follow. First, the fix is capacity, and the target is 70%, because at 70% it takes a 21% traffic increase to double p99 while at 90% it takes 5.6%. Second, before I add machines I would check whether service time itself is bimodal — one 2-second endpoint occupying the pool makes every other endpoint look slow, and that is a queue problem you solve by moving the slow work off the request path, not by buying servers.

“Do you need a message queue?” Testing: Little’s law. Only if some request holds a thread far longer than the rest, and here it does. Photo posts are 1% of traffic at 69 per second, and each holds a thread for a full second doing five resizes — that is 69 threads of concurrency, against 56 for the entire remaining 99% of traffic. So the slow 1% needs more of the pool than everything else combined, and worse, it blocks the fast requests behind it. Enqueueing instead costs 1 ms, so 0.07 threads. The second thing I get is burst absorption: a 10x photo burst for a minute builds a 33,000-job backlog that drains in 8 minutes, which converts an outage into a delay in thumbnail availability. What I would monitor is backlog divided by drain rate, in seconds — throughput dashboards look healthy right up to the point where the backlog is unbounded.

“How many nines can you promise?” Testing: whether you know why models overpredict. If instance failures were independent, 30 instances across 3 AZs would give more nines than anyone can measure. Real services with this shape land near 99.99%, which is 4.3 minutes a month, and the entire gap between the model and reality is correlated failure: one bad deploy, one AZ, one config push, one expired certificate. So the engineering that actually buys nines is not more replicas — it is rolling deploys with bake time, one AZ at a time, config changes that are versioned and revertible in seconds, and a permanent single-region fallback path. And I would state the error budget rather than the nines: 99.9% at our volume is 6 million failed requests a month, and today deploys alone would spend 60% of it.

Cheat sheet

This is the whole chapter compressed to the sentence you would say for each decision, meant for the last five minutes before an interview rather than for a first read.

Next: 02 — Back-Of-The-Envelope Estimation — the arithmetic underneath every number on this page, drilled until it is reflex.