17 — LLM Inference Performance

Every serving decision — batch size, scheduler, parallelism layout, even which GPU to buy — is downstream of one ratio: how much arithmetic the chip gets to do per byte it pulls from its own memory. The kv cache the most important mechanism in this chapter derived that decode is memory-bound and Prefill vs decode the fact underneath every cost rule priced the consequence at the API level. This chapter goes below both, to the silicon, where the same fact becomes a number you can compute — and then walks the full serving stack that exists to fight it: PagedAttention, continuous batching, chunked prefill, tensor and pipeline parallelism, disaggregated serving, and the profilers that tell you which one you need.

None of it requires GPU experience. It requires the kind of reasoning you already do when you notice a service is capped by disk reads rather than CPU — the hardware is different, the discipline is identical. Every claim below is derived from arithmetic you can reproduce with a calculator, because in the interview room a derived number survives follow-up questions and a memorized one does not.

By the end you should be able to:

• Place prefill and decode on a roofline plot and derive, not recite, why batch-1 decode runs at ~0.3% of the GPU’s peak.
• Size a KV cache from four model dimensions, and turn free memory into a max-concurrency number.
• Explain what PagedAttention borrows from OS virtual memory and compute what it buys.
• Say when continuous batching is not enough and chunked prefill earns its ITL/TTFT trade.
• Choose tensor vs pipeline parallelism from the communication arithmetic, not from folklore.
• Describe disaggregated serving and KV-aware routing as responses to the two phases having different bottlenecks.
• Read an Nsight Systems timeline and an Nsight Compute report, and say why nvidia-smi’s “GPU util” told you nothing.

Assumptions, fixed once. Unless a line says otherwise, every number below assumes an NVIDIA H100 SXM with 80 GB of HBM at 3.35 TB/s, 989 TFLOPS of dense BF16 compute, 900 GB/s of NVLink, and 64 GB/s of PCIe Gen5 x16 — representative spec-sheet values; real SKUs vary (an H200 has more and faster memory, a PCIe-card H100 less of both), and none of the reasoning changes when they do. BF16 means 2 bytes per number. Two reference model shapes recur: an 8B (8 × 10⁹ parameters: 32 layers, hidden width 4,096, 8 KV heads, head dimension 128) and a 70B (80 layers, hidden width 8,192, 8 KV heads, head dimension 128) — both Llama-3-shaped, chosen so the arithmetic checks against public configs.

1. The memory hierarchy, and why LLM serving is a bandwidth problem

A GPU is not a faster CPU. It is a machine built around one bet: that your problem is thousands of identical arithmetic operations wide, so it can trade single-thread speed for tens of thousands of simple arithmetic units running in lockstep.

Two words before anything else, because everything in this chapter uses them. A kernel is one function launched onto the GPU — “multiply these two matrices,” “apply softmax to these rows” — executed by thousands of threads at once; everything a GPU ever does is a sequence of kernel launches, and a serving engine’s decode step is a train of them. An SM (streaming multiprocessor) is the unit those threads are scheduled onto: the H100 SXM has 132 of them, and each SM holds the arithmetic units, a slice of fast on-chip memory, and the scheduler for its resident threads. Threads execute in groups of 32 called a warp — one instruction, 32 data lanes, which is the lockstep.

The design has one more consequence worth knowing before the memory pyramid. A CPU hides memory latency with big caches and speculation; a GPU hides it with more parallelism. Each SM keeps many warps resident at once, and the moment one warp stalls waiting for a memory read, the scheduler issues instructions from another. That trick works only while there are enough independent operations in flight — hold that thought, because batch-1 decode is precisely the workload that fails to supply them.

The pyramid

All the arithmetic in the world is worthless if the data isn’t next to it. The memory system is a pyramid, and each level trades capacity for speed:

Three of those rows deserve one sentence each. SRAM is static RAM on the die itself — the only memory fast enough to keep the arithmetic units fed, and four orders of magnitude too small to hold a model. HBM is the stack of DRAM sitting on the GPU package: its 80 GB is the capacity in every “does it fit” conversation, and its 3.35 TB/s is the number this whole chapter orbits. NVLink is NVIDIA’s GPU-to-GPU interconnect inside a server — 3.7× slower than HBM but 14× faster than PCIe, the general-purpose bus to the host, and that 14× gap decides where multi-GPU schemes are allowed to live (§7).

flowchart TD
    R["Registers<br/>KBs per SM, ~1 ns"] -->|"within the SM"| S["SRAM: shared mem + L2<br/>~50 MB L2, tens of TB/s"]
    S -->|"on-die"| H["HBM<br/>80 GB @ 3.35 TB/s"]
    H -->|"NVLink ~900 GB/s"| G["Other GPUs in the node"]
    H -->|"PCIe ~64 GB/s"| D["Host DRAM / NVMe"]

    style H fill:#2d6a4f,color:#fff
    style G fill:#40916c,color:#fff
    style D fill:#bc6c25,color:#fff

The ratio that runs the chapter

Now put the compute next to the memory. A FLOP is one floating-point operation — one multiply or one add — and FLOP/s (often written FLOPS) is how many of them per second. The H100 does 989 × 10¹² of them per second in dense BF16. Divide the two headline numbers and you get the chip’s personality in one line:

peak compute          = 989e12 FLOP/s
HBM bandwidth         = 3.35e12 bytes/s
ratio                 = 989e12 / 3.35e12  =  295 FLOPs per byte

For every byte the chip fetches from HBM, it has time to do ~295 floating-point operations before the memory system becomes the reason it waits. Feed it fewer than 295 FLOPs of work per byte and the arithmetic units idle; the workload is memory-bound — throughput set by bytes moved, not by math. Feed it more and it is compute-bound — the arithmetic units are the wall, and bandwidth is no longer the story.

This is the same shape as capacity-planning a service where every request scans data from disk: past a point, adding CPU does nothing, because requests per second equals disk bandwidth divided by bytes per request. If you have ever said “we’re I/O-bound, more cores won’t help,” you already own the reasoning — on a GPU serving LLM decode, the “disk” is HBM, and, as §2 will show, the bytes per request are enormous.

Two immediate consequences of the table, before the roofline formalizes them.

First, a floor you can compute in your head: touching everything in HBM once — which a decode step approximately does — takes

80 GB / 3.35 TB/s = 80e9 / 3.35e12 = 24 ms

so any workload that must sweep the whole card’s memory per step is capped at ~40 such steps per second, no matter what the arithmetic units do. Keep that number; §4 fills the 80 GB with weights and KV cache and the cap becomes a token rate.

Second, one famous kernel makes the hierarchy concrete. FlashAttention’s entire contribution is refusing to write the attention score matrix to HBM: it tiles the computation so scores are produced, used, and discarded inside SRAM, trading extra arithmetic for eliminated memory traffic. Extra FLOPs to save bytes is a good trade on a chip that has 295 FLOPs to spend per byte — that one sentence is the design philosophy of every fast kernel you will profile in §9.

Why you care: almost every phrase in an inference job description — batching, paged KV, quantization, speculative decoding, disaggregation — is a scheme for changing the FLOPs-per-byte of decode, or for spending the FLOPs that decode leaves idle. Hold the 295 and the rest of the chapter is derivable.

2. Roofline modeling

One plot organizes everything: the roofline. It answers, for any kernel or workload, “what throughput is this allowed to reach on this chip?” — before you profile anything, from two numbers you can look up and one you can count.

Define the x-axis first. Arithmetic intensity is the FLOPs a workload performs per byte it moves from HBM:

intensity = FLOPs performed / bytes moved from HBM      units: FLOP per byte

The roofline is then a ceiling with two segments:

attainable FLOP/s = min( peak compute ,  intensity × bandwidth )

At low intensity the second term is smaller: you are paying for bytes, and doubling intensity doubles attainable throughput — a rising slope whose gradient is the memory bandwidth. At high intensity the first term caps you: the flat compute roof. The corner where they meet is the ridge point, and it is the 295 from §1, computed properly:

ridge = peak compute / bandwidth
      = 989e12 FLOP/s / 3.35e12 B/s
      = 295 FLOP per byte

One honesty note on the numerator, because it is the most common way to get this wrong by 2×. NVIDIA’s headline figure for the H100 is 1,979 TFLOPS BF16 — but that assumes 2:4 structured sparsity, a format where two weights in every group of four are zero and skipped by the hardware. Served LLM weights are dense; nothing in a standard checkpoint has that structure. So the honest peak is the dense number, 989. Quoting the sparse figure silently halves your ridge point and doubles your apparent MFU (§9), and an interviewer who serves models for a living will hear it immediately.

 attainable                SCHEMATIC - log-log axes, shape only
 FLOP/s
   989 T |. . . . . . . . . . . _______________●________   compute roof
         |                     /.               prefill,
         |                    / .               2,048-token prompt
         |        memory     /  .
         |        roof:     /   .
         |        slope =  /    .
         |        3.35TB/s/     .
  3.35 T |. . ●. . . . . /      .
         |  decode,     /       . ridge = 295
         |  batch 1    /        .
         +----1--------------- 295 ------- 2,048 ---------
                    arithmetic intensity, FLOP per byte

Placing decode on it

Generating one token means multiplying activations through essentially every parameter of the model. Two facts give the intensity directly.

First, the FLOP count. A matrix multiply against N parameters does one multiply and one add per parameter — a multiply-accumulate — so a forward pass costs ~2 FLOPs per parameter per token. (Attention over the KV cache adds more on long contexts; ignoring it makes decode look better than it is, so the conclusion below survives the simplification.)

Second, the byte count. Each BF16 parameter is 2 bytes, and at batch 1 every one of them must be streamed from HBM for every single token — 16 GB of weights cannot be cached in 50 MB of SRAM, so nothing carries over from one token to the next.

decode intensity (batch 1) = 2 FLOPs per parameter / 2 bytes per parameter
                           = 1 FLOP per byte

(Worth asking why the denominator is 2 and not 4. Models are served in 16-bit — BF16 keeps FP32’s exponent range in half the bytes — because on the memory roof, bytes are throughput: serving in FP32 would double every byte count in this chapter and halve every decode ceiling, for accuracy nothing downstream can measure. Quantization in §10 is the same logic continued past 16.)

One. Against a ridge of 295. Batch-1 decode sits at the extreme left of the plot, and the roofline prices the damage:

attainable = min( 989e12 , 1 × 3.35e12 )  =  3.35e12 FLOP/s
fraction of peak = 3.35e12 / 989e12       =  0.34%

The GPU runs at a third of one percent of its rated compute. Not because anything is misconfigured — because the workload physically cannot feed the arithmetic units. Check it in the time domain on the 8B reference model, one line per step:

weights                = 8e9 params × 2 B          = 16 GB
time to stream them    = 16e9 / 3.35e12            = 4.78 ms per token
decode ceiling         = 1 / 0.00478               = 209 tokens/s
FLOPs per token        = 2 × 8e9                   = 16 GFLOP
time to compute them   = 16e9 / 989e12             = 16.2 µs
waiting-to-working     = 4,780 µs / 16.2 µs        = 295 : 1

The 295 reappearing is not a coincidence — it is the ridge point measured in time instead of intensity: 295 microseconds of waiting on memory for every microsecond of arithmetic. This is also §1’s latency-hiding machinery failing for lack of material: one token’s worth of work simply does not contain enough independent operations to keep 132 SMs busy while 16 GB streams past. (Chapter 00 §4 computed the same ceiling from the KV-cache side at 100K context; at short contexts the weights are the dominant stream, and the two calculations meet in §4.)

Placing prefill on it

Prefill processes the whole prompt in one pass, so each weight fetched from HBM is used against every prompt token before it is discarded:

prefill intensity ≈ 2 FLOPs × n_tokens per parameter / 2 bytes per parameter
                  = n_tokens FLOP per byte

2,048-token prompt: intensity ≈ 2,048  >>  ridge 295   ->  compute-bound

Same model, same weights, same GPU — opposite side of the ridge. And the roofline prices this side too:

prefill FLOPs (8B, 2,048 tokens) = 2 × 8e9 × 2,048   = 32.8 TFLOP
at the full 989 TFLOPS roof      = 32.8e12 / 989e12  = 33 ms
at a realistic 50% of peak       = 32.8e12 / 494e12  = 66 ms

(No real workload sits on the roof — attention, normalization, and kernel-launch overhead drag it down; 40-60% of peak is a strong prefill, a claim §9 turns into the measurable MFU. The 50% here is an assumption, flagged and reused consistently.)

One boundary case falls out of the formula and surprises people: prefill intensity is ≈ the prompt length in tokens, so a prompt shorter than the ridge point — under ~295 tokens — is still on the memory roof. A fleet serving short classification prompts never sees compute-bound prefill at all; “prefill is compute-bound” is a claim about prompts long enough to amortize the weight-read, not a law of nature. The formula tells you where your own workload sits; the slogan doesn’t.

Prefill saturates arithmetic; decode starves it. One model, two phases, two different machines — which is §3’s subject, and the single most useful diagram to draw unprompted in any inference interview.

Batching moves decode up the roofline

The batch-1 disaster has one clean exit: make each weight-read serve more than one token. Decode B sequences together — one new token for each of B users in the same forward pass — and each streamed parameter contributes to B token-generations:

decode intensity at batch B ≈ B FLOP per byte
break-even with the ridge:    B ≈ 295

So decode needs a batch in the hundreds before the compute roof is even visible. Two caveats keep this honest, and both matter later.

First, the KV cache does not amortize. Weights are shared across the batch; each sequence’s KV cache is its own, and every step streams all of it. Write the intensity with both terms and watch what long contexts do:

effective intensity = B × 2P FLOPs / ( 2P + B × kv_bytes ) bytes
                      where P = params, kv_bytes = per-sequence live cache

8B, B = 64, short chats (600 live tokens, 78.6 MB each — §4):
    bytes = 16e9 + 64 × 78.6e6 = 21.0e9
    intensity = 64 × 16e9 / 21.0e9              = 49    (not 64)

8B, B = 64, long contexts (8,000 tokens, 1.05 GB each):
    bytes = 16e9 + 64 × 1.05e9 = 83.2e9
    intensity = 64 × 16e9 / 83.2e9              = 12    (not 64)

Same batch, a quarter of the intensity, purely because the caches got long. Long-context serving is memory-bound twice over — capacity and bandwidth — which is why §4 sizes the cache so carefully and why KV-shrinking tricks compound.

Second, holding hundreds of sequences resident is a memory-capacity problem: every one of them needs its KV cache in HBM simultaneously. That is why §5’s memory manager — a thing that sounds like janitorial software — is one of the largest throughput features ever shipped for LLM serving.

Batching is the master lever; everything in §§4-6 exists to let you pull it further.

Interview move: when any performance question lands, draw the roofline first — two axes, two roofs, the ridge at ~295, decode at 1, prefill at prompt-length. Then place the question on it. “Should we quantize?” is “move decode right by shrinking bytes.” “Why is util high but throughput low?” is “you are on the memory roof; util doesn’t measure that.” One diagram, every follow-up becomes navigation.

3. Prefill and decode are two different machines

The user-visible metrics split exactly along the phase boundary, so name them precisely:

• TTFT — time to first token: queueing plus the entire prefill. The “how long do I stare at nothing” number.
• ITL — inter-token latency: the gap between one streamed token and the next, as observed per token. Its spikes are what users feel as stutter.
• TPOT — time per output token: the mean of the ITLs over a response. TPOT summarizes; ITL’s distribution diagnoses — a clean mean with a terrible p99 is a specific disease with a specific cause, coming two paragraphs down.

(Latency budgets a whole request in these terms from the client side; here the concern is what one GPU does to them when it serves both phases at once.)

Put ordinary numbers on both, 8B model, using §2’s arithmetic. A 2,048-token prompt prefills in ~66 ms at the assumed 50% of peak, so TTFT lands well under typical targets before queueing. Decode at a healthy batch runs a step every ~6 ms (derived in §5), so a smooth stream shows ITL ≈ 6 ms. Representative SLOs for chat — TTFT p90 under ~500 ms, ITL p99 under a few tens of ms — are comfortably met. On paper.

Budget a whole request from those two numbers and notice where the time goes:

TTFT                        =  66 ms      (prefill, 2,048 tokens)
decode, 250 output tokens   = 250 × 6 ms  = 1,500 ms
total                       ≈ 1.57 s
decode share                = 1,500 / 1,566 = 96%

Ninety-six percent of the request’s wall clock is decode — the phase running at 0.3% of the chip’s arithmetic. That is Latency “decode is 89% of it” observation, re-derived from the silicon side, and it is why every latency lever that matters (§10) attacks decode: shortening the prompt attacks the 4%.

Head-of-line blocking

A serving GPU never runs one request. At any instant it holds a batch of in-flight decodes — each wanting its small, regular ~6 ms heartbeat — and a queue of new arrivals, each wanting a prefill that is a solid slab of compute. The two workloads share one set of SMs, and a naive scheduler runs whole requests at a time. Then a long prompt walks in:

in-flight decode step                               ≈ 6 ms    -> smooth ITL ≈ 6 ms
new arrival: 8,000-token prompt
prefill FLOPs      = 2 × 8e9 × 8,000               = 128e12 FLOP
prefill time       = 128e12 / (0.5 × 989e12)       = 259 ms

every in-flight decode stalls for ~260 ms          -> worst ITL ≈ 43× baseline

One long prompt turns every concurrent user’s smooth stream into a quarter-second freeze. That is head-of-line blocking — a long job in a shared queue delaying every short one behind it — and it is why the ITL distribution diagnoses: the mean barely moves, while p99 explodes in exact correlation with long-prompt arrivals.

Note carefully that priorities alone cannot fix it. “Decode always preempts prefill” just starves TTFT — arrivals queue behind an endless decode heartbeat and the newcomer waits forever for their first token. “Prefill first” is the 260 ms freeze as policy. The phases want opposite things from the same silicon, and no priority ordering gives both:

Read the bottom row twice, because it quietly says the two phases don’t even autoscale on the same signal: prefill demand grows with prompt traffic, decode demand with concurrent streams, and a workload shift (longer documents, chattier sessions) moves them independently.

Hold the tension; it resurfaces twice. §6’s chunked prefill makes the two phases share one GPU politely — a scheduling answer. §8’s disaggregation stops making them share at all — an architecture answer. Which one you need is a fleet-size question, not a taste question.

4. The KV cache, sized honestly

Chapter 00 §4 derived why the KV cache exists and what decode pays to re-read it; this section treats it purely as a capacity problem, because on a serving GPU the KV cache is the thing that decides your batch size — and §2 just made batch size the master lever.

The size formula, term by term. Attention stores one K vector and one V vector per token — that is the leading 2. It does so independently in every layer, because each layer runs its own attention (Attention and why context costs what it does is the mechanism). Within a layer, one K/V pair per KV head, each head_dim numbers wide, each number dtype_bytes wide:

bytes per token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

The reason n_kv_heads is its own factor rather than “the number of heads”: in GQA (grouped-query attention) several query heads share one K/V pair, precisely so this number is smaller than the query-head count. MQA (multi-query attention) is the limit case — every query head shares a single K/V pair. Both are architecture decisions made at training time whose entire purpose is visible in this serving-time formula.

Substitute the 70B shape (80 layers, 8 KV heads via GQA, head_dim 128, FP16 = 2 bytes):

bytes per token = 2 × 80 × 8 × 128 × 2       = 327,680 B  ≈ 320 KB
8K sequence     = 327,680 × 8,000            = 2.62 GB     (8K taken as 8,000 tokens)

A third of a megabyte per token; two and a half gigabytes per long chat. And the formula is linear in context, so stretch it:

The last row is worth saying out loud: one 128K-context conversation eats more than half an H100’s entire memory in cache alone. Long-context serving is not “the same thing with bigger numbers” — it is a regime where a handful of requests exhaust the card and §2’s batching lever barely moves.

Why 70B needs tensor parallelism before anything else

Now try to place the model itself on one card:

weights = 70e9 × 2 B = 140 GB    vs    HBM = 80 GB

The weights alone are 1.75× the card. Before any KV cache, any activation, any batch, a 70B model in 16-bit cannot exist on one H100. That sentence — fit, not speed — is the first reason 70B-class serving is tensor-parallel (§7). At TP = 4 the weights shard to 35 GB per GPU and the KV cache shards with the heads (8 KV heads / 4 GPUs = 2 per GPU):

per GPU:   80 − 35 (weights) − 4 (runtime reserve)  = 41 GB for KV
aggregate: 4 × 41                                    = 164 GB
concurrent 8K sequences ≈ 164 / 2.62                 ≈ 62

Max concurrency, worked on the model that does fit

The 8B model on a single H100, same formula, every substitution written out:

bytes per token   = 2 × 32 × 8 × 128 × 2      = 131,072 B  = 128 KB
per-seq KV at 8K  = 131,072 × 8,000           = 1.05 GB

weights           = 8e9 × 2 B                 = 16 GB
runtime + activations reserve                 ≈  4 GB
free for KV       = 80 − 16 − 4               = 60 GB

max concurrent    = 60 / 1.05                 ≈ 57 sequences

Read that against §2’s break-even: decode wants a batch near 295 to reach the ridge, and a full card of worst-case 8K sequences supports 57. Even with memory packed wall to wall, decode stays memory-bound. That is the punchline of the whole section — the batching lever runs out of memory long before it runs out of usefulness, so every byte shaved off the KV cache converts directly into batch, and batch directly into throughput.

Which is why KV-shrinking choices get made at model-design time, before a serving engineer ever touches the thing. Same 70B shape under the three attention designs:

GQA is an 8× cache reduction taken before a single serving trick runs — a 70B with MHA would hold three 8K conversations per 80 GB of cache space; with GQA it holds twenty-four. The serving-time equivalent — quantizing the KV cache to FP8, halving dtype_bytes — buys another 2× from the same formula, and stacks.

Where this chapter’s ceiling meets chapter 00’s

Two decode ceilings are now in circulation and they look inconsistent: §2 derived 209 tok/s counting only the weights; chapter 00 §4 derived ~102 tok/s counting only the cache. Neither is wrong — each ignored the other’s stream, and a real decode step pays both:

step time = ( weight bytes  +  live KV bytes ) / bandwidth

Run it for the 8B at batch 1 as the context grows, one line per regime:

    1,000 tokens:  16 + 0.13 GB = 16.1 GB  ->  16.1 / 3,350 GB/ms = 4.8 ms  ->  208 tok/s
   10,000 tokens:  16 + 1.3  GB = 17.3 GB  ->  17.3 / 3,350       = 5.2 ms  ->  193 tok/s
  100,000 tokens:  16 + 13.1 GB = 29.1 GB  ->  29.1 / 3,350       = 8.7 ms  ->  115 tok/s

Short contexts: the weights dominate and §2’s ceiling holds. Long contexts: the cache takes over and chapter 00’s number is the truer one. The crossover — where one conversation’s cache costs as much bandwidth as the entire model — sits where the two streams are equal:

16e9 bytes / 131,072 bytes per token ≈ 122,000 tokens

One 122K-token context doubles the cost of every token this sequence will ever generate. Per-sequence decode cost is not flat; it degrades with context length even after you have paid the memory to hold it — the silicon-level reason long conversations feel slower, and the second reason (after §2’s intensity drag) that compaction and context discipline are performance features, not just cost ones.

Why you care: “size the KV cache for model X” is a standard screen, and the trap inside it is the head count. Reciting 64 heads for a GQA model inflates the answer 8×. The safe answer names the formula, asks (or states) whether the model uses GQA, and substitutes out loud.

5. PagedAttention — virtual memory for the KV cache

The §4 arithmetic quietly assumed every allocated KV byte holds a real token. In the serving systems before 2023, most of them didn’t — and fixing that one assumption produced the largest single throughput win in the modern stack.

The fragmentation problem

A sequence’s KV cache grows one token at a time toward a length nobody knows in advance — the model stops when it emits a stop token, and users cancel. The attention kernels of the day required each sequence’s cache to be contiguous in memory. The only safe contiguous allocation for an unpredictable length is the worst case: reserve max_len up front, per request, the moment it arrives.

reservation per request (8B, max_len 8,192)  = 8,192 × 131,072 B = 1.07 GB
a chat turn that actually uses 300 tokens    =   300 × 131,072 B = 39 MB
internal fragmentation                        = 1 − 39 MB / 1.07 GB ≈ 96%

That reserved-but-unused space is internal fragmentation — memory allocated to a request and holding nothing, the same pathology as a fixed-size buffer pool sized for the largest message. It was not the only leak. The pre-paged systems wasted KV memory three distinct ways:

• Internal fragmentation — the worst-case reservation above, by far the biggest term.
• External fragmentation — variable-sized contiguous slabs leave unusable gaps between allocations, exactly like a heap without a compactor.
• Duplication — sequences sharing a prefix (same system prompt, same few-shot header) each stored their own identical copy of its K/V.

Across realistic traffic the vLLM paper measured 60-80% of KV memory wasted between the three. The GPU wasn’t short of memory; it was short of usable memory, and batch size — the master lever — was capped by the waste.

The fix, borrowed whole from operating systems

PagedAttention is the vLLM project’s application of the oldest trick in OS design — virtual memory — to the KV cache:

• Carve KV memory into fixed-size blocks of, say, 16 tokens each (the page).
• Give each sequence a block table: a small array mapping its logical token positions to whichever physical blocks it happens to own — no contiguity required (the page table).
• Allocate a new block only when the previous one fills — allocation on demand, not on arrival.

The attention kernel walks the block table the way a CPU’s memory unit walks page tables:

sequence A, 40 tokens live, block size 16:

logical tokens   0-15    16-31    32-39 and growing
physical block     7       12        3
block 3 is 8/16 full -> the ONLY waste this sequence has

Fragmentation collapses to at most one partially-filled block per sequence:

worst-case waste per sequence = 16 tokens × 131,072 B ≈ 2 MB    (vs ~1 GB reserved)

The block size is itself a small trade, and it is the same one page size is in an OS. Smaller blocks bound waste tighter but make block tables longer and KV reads more scattered — more indirection per attention step; bigger blocks read more contiguously but let internal fragmentation creep back (a 1,024-token block wastes up to 1,023 tokens per sequence, and you are halfway back to the reservation model). Sixteen-ish tokens is where the implementations landed, and “it’s a page-size trade” is the sentence that shows you understood the borrowing rather than memorized it.

Two more OS ideas come along free, and they kill the duplication term. Sequences sharing a prefix point their block tables at the same physical blocks — prefix sharing, one copy of the system prompt no matter how many users are behind it. And a shared block is copied only at the moment someone writes into it: copy-on-write, exactly the fork semantics. One 2,000-token system prompt at 100 concurrent users:

naive:  100 × 2,000 × 131,072 B = 26.2 GB of identical bytes
paged:    1 × 2,000 × 131,072 B = 0.26 GB

(When memory still runs out — a burst of long generations — the engine preempts: it evicts a sequence’s blocks and later either recomputes them or restores them from a copy swapped to host DRAM over PCIe. vLLM exports preemption counters; a nonzero rate is your signal that the KV budget, not the scheduler, is the binding constraint.)

What it buys, in tokens per second

The point was never memory hygiene; it was batch. Same 8B card, same 60 GB of KV space, a mixed workload averaging 600 live tokens per sequence:

live KV per sequence = 600 × 131,072 B                    = 78.6 MB

naive batch  = 60 GB / 1.05 GB reservation                ≈ 57 sequences
paged batch  = 60 GB / 78.6 MB actual                     ≈ 760 sequences

Careful with the next step, because the honest version is subtler than “13× the batch, 13× the throughput.” Wasted space is not wasted traffic — the naive system never reads its empty reservations, so both systems stream only live bytes. What changes is how many sequences share each pass over the weights:

per-step HBM traffic = weights + live KV:
naive:  16 + 57 × 0.0786   = 20.5 GB  -> 20.5 / 3,350 GB/ms = 6.1 ms  ->  57 / 0.0061  ≈  9,300 tok/s
paged:  16 + 760 × 0.0786  = 75.7 GB  -> 75.7 / 3,350 GB/ms = 22.6 ms -> 760 / 0.0226  ≈ 33,600 tok/s

3.6× the throughput from the same silicon, squarely inside the 2-4× the vLLM paper reported over the systems it compared against — and the mechanism is exactly §2’s: more sequences amortizing each weight-read, intensity climbing the memory roof. Note the fine print the arithmetic exposes: ITL rose from 6.1 ms to 22.6 ms, because a bigger batch makes every step heavier. Throughput and per-token latency trade against each other, and production schedulers cap batch to hold an ITL target rather than maximizing tokens per second — the reason a serving config has a knob there at all.

vLLM is the reference implementation and the default open-source serving stack; PagedAttention has since been adopted essentially everywhere — TensorRT-LLM, SGLang, and the rest — to the point where “the KV cache is paged” is now an assumption, not a feature.

What to say you measured: standing this up is a weekend, not a research project. Serve an 8B model with vLLM, sweep --max-num-seqs, and plot tokens/s and p99 ITL against batch while watching the KV-cache-usage and preemption counters it exports. What you will see — throughput climbing near-linearly with batch, then flattening as KV memory saturates, ITL rising the whole way, preemptions appearing right where the curve bends — is §2’s roofline and §4’s capacity wall traced empirically. In the interview, “I ran the sweep and watched the knee appear” converts every claim in this chapter from vocabulary into evidence, and it costs one Saturday.

6. Batching strategies

Batching is the lever; the strategy question is when requests are allowed to join and leave the batch. Three generations of answer, each fixing the previous one’s idle time.

Static batching: wait, pad, drain

Static batching is the inheritance from the translation-model era: collect B requests, pad them to equal length, run the whole batch to completion, admit nobody until it drains. It assumes what was true for translation — outputs of similar, predictable length — and LLM traffic violates it maximally: generation lengths vary by two orders of magnitude and are unknown in advance. The batch runs at the pace of its longest member while finished slots idle:

batch of 8, output lengths 100, 200, ..., 800 tokens
slot-steps run    = 8 × 800   = 6,400
slot-steps useful = 100 + 200 + ... + 800 = 3,600
utilization       = 3,600 / 6,400 = 56%

Nearly half the decode capacity spent computing padding — the tail of every batch is a mostly-empty machine — plus a second, quieter cost: arrivals queue until the entire batch drains, so one 800-token response holds the door shut for seconds. Head-of-line blocking again, this time at the batch boundary.

Continuous batching: the iteration is the scheduling unit

Continuous batching is the fix, introduced by the Orca paper under the name iteration-level scheduling. The insight is that decode has a natural preemption point every few milliseconds — the token boundary — because every sequence in the batch is doing the identical thing: one forward pass, one token. So stop scheduling requests and start scheduling iterations: after every single decode step, finished sequences exit the batch and waiting requests join it. A slot that frees at token 100 is serving a new user at token 101.

flowchart LR
    subgraph ST["Static - slots wait for the longest request"]
        S1["step 1<br/>A B C D"] --> S2["step 150<br/>A · C ·"] --> S3["step 700<br/>A · · ·"] --> S4["drain, then<br/>admit E F G H"]
    end
    subgraph CB["Continuous - slots refill at token boundaries"]
        C1["step 1<br/>A B C D"] --> C2["step 150<br/>A E C F"] --> C3["step 700<br/>G E H F"] --> C4["never drains"]
    end

    style S2 fill:#9d0208,color:#fff
    style S3 fill:#9d0208,color:#fff
    style C2 fill:#2d6a4f,color:#fff
    style C3 fill:#2d6a4f,color:#fff

Utilization stops depending on length variance entirely — the batch stays full as long as the queue is nonempty — and the padding waste disappears with it. The same move also deletes the batch-boundary queueing: a new arrival waits at most one iteration (milliseconds) for admission instead of one batch-drain (seconds), which is a TTFT win the utilization number doesn’t even show. Notice also why this and §5 ship together: join-anytime scheduling needs allocate-anytime memory. A scheduler admitting a sequence mid-flight cannot pre-reserve a contiguous gigabyte; paged KV is what makes iteration-level admission affordable. Every modern engine — vLLM, TensorRT-LLM, SGLang — runs this pair by default, which is why “we use continuous batching” is table stakes in a deep-dive, not a differentiator.

Chunked prefill: the knob and its trade

Continuous batching still gets one thing wrong, and §3 already computed it: joining the batch requires a prefill, and a long prefill freezes every in-flight decode for ~260 ms. The scheduler has two bad options — run the prefill now and spike everyone’s ITL, or defer it and let TTFT rot in the queue.

Chunked prefill (the Sarathi lineage; now a standard engine option) dissolves the dilemma by splitting the prompt into fixed-size chunks — say 512 tokens — and co-scheduling one chunk per iteration alongside the normal decode step, instead of letting prefill monopolize the GPU:

8,000-token prompt, 512-token chunks               = 16 chunks
chunk compute = 2 × 8e9 × 512 / (0.5 × 989e12)     = 16.6 ms
iteration     = decode step 6.1 + chunk 16.6       ≈ 23 ms

worst ITL during a long admission:  260 ms  ->  ~23 ms    (11× better)
TTFT for the long prompt: 16 chunks × 23 ms ≈ 370 ms  vs  ~260 ms dedicated  (+40%)

That is the knob and its trade in two lines: chunk size buys ITL smoothness for the crowd at a TTFT cost to the long-prompt request. Small chunks favor streaming chat; big chunks favor TTFT-sensitive, long-document workloads. In engine configs the same idea usually surfaces as a token budget per iteration — one number capping decode tokens plus prefill-chunk tokens per step, which is the knob you actually turn.

The deeper point, worth saying in an interview: chunked prefill deliberately lowers the efficiency of prefill (each chunk re-reads the weights that a monolithic prefill would have read once) to protect a latency SLO. It is a page taken from every soft-real-time system ever built — smooth beats fast — and when no chunk size satisfies both sides of the trade, that is the cue for §8.

The three strategies, as one decision:

7. Tensor and pipeline parallelism

§4 left a fact on the table: 140 GB of 70B weights cannot occupy an 80 GB card. Multi-GPU inference is not an optimization — for big models it is the entry ticket. The two ways to split a model differ in what they cut, and the communication arithmetic — not folklore — decides which to use where.

Clear one distributed-systems reflex out of the way first: “just add replicas” is not on this menu. Replication — the same full model on more GPUs behind a load balancer — scales request throughput and is exactly right once the model fits; it does nothing for a model that doesn’t fit and nothing for per-token latency, because each replica still faces §2’s memory wall alone. TP and PP are model parallelism: they split one copy of the model itself, which is why they exist and why they cost communication.

Tensor parallelism: cut every matrix

Tensor parallelism (TP) splits each weight matrix across p GPUs — attention heads dealt out (8 KV heads at TP = 4 means 2 per GPU, and the KV cache shards along with them), MLP columns likewise — so every GPU computes a slice of every layer, in lockstep, for every token.

Partial results must be recombined, and the count is exactly two per layer. The standard Megatron-style layout splits the first matrix of a pair column-wise and the second row-wise, so the intermediate never needs communicating — but the output of each pair is a partial sum that every GPU needs in full. There are two such pairs per transformer layer (the attention block ending in its output projection; the MLP ending in its down-projection), hence two all-reduce operations — the collective where every GPU ends up holding the sum of all GPUs’ contributions — per layer, per token. The library doing these collectives is NCCL (NVIDIA’s collective-communication library), whose kernels you will meet by name on the §9 timeline.

Per token, the payload of each all-reduce is one activation vector, so communication scales with the hidden dimension. The 70B shape at TP = 4, decoding, one line per step:

activation per token = 8,192 × 2 B                 = 16 KB
ring all-reduce traffic per GPU ≈ 2(p−1)/p × 16 KB = 24 KB
collectives per token = 2 × 80 layers              = 160
bytes per token per GPU = 160 × 24 KB              = 3.8 MB
wire time on NVLink     = 3.8e6 / 900e9            ≈ 4 µs

The bytes are trivial — that is worth pausing on, because it surprises people. What is not trivial is that a decode step issues 160 synchronizing collectives, and each one carries a fixed launch-plus-latency floor of a few microseconds regardless of payload. Count latency, not bytes:

latency floor ≈ 160 × 5 µs                         = 0.8 ms per token
decode step at TP = 4: weights 35 GB / 3.35 TB/s   = 10.4 ms
overhead on NVLink                                 ≈ 8%

same 160 collectives over PCIe (~25 µs each)       ≈ 4 ms  ->  ~40% overhead

That last line is the whole placement rule: TP is chatty per token, so it lives on NVLink, inside one node. Stretch it across a slower link and the collectives eat the step. In exchange, TP is the only scheme that reduces per-token latency: each GPU streams 1/p of the weights, so §2’s memory wall divides by p —

70B, batch 1, TP = 4:  35 GB / 3.35 TB/s = 10.4 ms  + 0.8 ms comm  ≈ 11.2 ms
decode ceiling                                                     ≈ 90 tok/s

— for a model that, one section ago, could not exist on one card at all.

Pipeline parallelism: cut the layer stack

Pipeline parallelism (PP) assigns each GPU a contiguous run of layers — a stage — and tokens flow through the stages like a factory line: 80 layers at p = 4 is 20 layers per stage. Communication per stage boundary is one 16 KB activation handoff, three boundaries total, once per token — bytes and message-counts so small the interconnect barely matters. PP is what crosses nodes.

The price has a name: the pipeline bubble. While stage 1 works on the first piece of work, stages 2-4 sit idle; symmetrically, stage 1 goes idle while the last work drains through. The cure is to keep the pipe full: split the batch into m microbatches — smaller waves fed in back-to-back so all stages overlap on different waves. The idle fraction is start-up plus drain over total slots, and it is worth deriving once because interviewers ask for it:

time to fill the pipe       = p − 1 stage-steps
useful work                 = m stage-steps per stage
total                       = m + p − 1

bubble fraction = (p − 1) / (m + p − 1)

p = 4, m = 16:   3 / 19 = 15.8% of stage-time idle
p = 4, m = 64:   3 / 67 =  4.5%

Worked once, remembered forever: the bubble dies only when microbatches vastly outnumber stages, which requires traffic. PP is therefore a throughput regime, not a latency one — a token still visits all 80 layers sequentially, so single-stream latency does not improve by a nanosecond, and a deep pipeline under thin traffic is mostly bubble.

Choosing, and combining

The standard layout follows directly: TP as far as the NVLink domain reaches (typically the 8 GPUs of one node), and PP across nodes on top only when a whole node is still too small. Work the flagship case:

405B model, BF16:  405e9 × 2 B = 810 GB of weights
one node, TP = 8:  810 / 8     = 101 GB per GPU   -> does not fit in 80
TP = 8 × PP = 2 (16 GPUs):  810 / 16 ≈ 51 GB per GPU  -> fits, with room for KV

Two nodes, TP inside each, PP between them — every placement decision in that line came from the arithmetic above, none from preference.

(The measurable version, if you extend §5’s weekend project: vLLM takes --tensor-parallel-size on a multi-GPU box. Serve the same model at TP=1 and TP=2 and watch batch-1 latency drop by nearly half while tokens-per-GPU throughput barely moves or dips — the collectives’ overhead made visible. That one contrast is the whole TP story in two runs.)

Training uses the same two words plus more — data parallelism, gradient all-reduces, optimizer sharding — but optimizes a different objective (step time over a fixed corpus, with gradients to synchronize) where inference parallelism only has to fit weights plus KV and hit a latency target; Scale covers that side, and conflating the two regimes is a known interview trap.

8. Disaggregated serving and KV-aware routing

Chunked prefill made the two phases share a GPU politely. The stronger move — once the fleet is big enough — is to stop making them share.

Disaggregated serving runs prefill and decode on separate GPU pools. A request lands on the prefill pool, which computes the TTFT-critical work as one uninterrupted compute-bound slab — no chunking, no compromise; the KV cache it produced ships to a decode-pool worker, which streams tokens from it with a heartbeat nothing ever interrupts. The idea comes out of the DistServe and Splitwise papers; Dynamo is NVIDIA’s productized version of the same lineage, and rack-level designs pair bandwidth-heavy SKUs with the decode pool for exactly the reasons §2 derived.

Why bother, when it adds a network hop? Because §3’s table ended on the observation that the phases scale on different axes, and a shared GPU forces one box to be provisioned for both. Separate them and each pool is sized, scheduled, and scaled against its own bottleneck:

• The prefill pool is compute-bound: size it in prompt-tokens-per-second, scale it when TTFT degrades.
• The decode pool is bandwidth- and memory-bound: size it in concurrent-sequences-at-target-ITL, scale it when ITL degrades.

Interference is gone by construction — there is no long prompt to stall a decode, because they are on different machines — and the pools autoscale independently. It is the same argument as splitting a monolith’s read and write paths onto separately-scaled replicas, and the same paper trail: DistServe frames the win as goodput — requests per second that meet both the TTFT and ITL SLOs, rather than raw tokens per second — and shows shared GPUs sacrificing goodput even at high utilization, because utilization doesn’t know which SLO you just blew.

Size the pools with arithmetic, not vibes. Representative traffic — 100 req/s, mean prompt 2,000 tokens, mean output 250 tokens — against representative per-pool capacities (labeled representative; measure your own):

prompt demand = 100 × 2,000 = 200,000 prompt tok/s
prefill capacity ≈ 10,000 tok/s per GPU group     -> 20 prefill groups

output demand = 100 × 250   = 25,000 output tok/s
decode capacity ≈ 2,500 tok/s per GPU group       -> 10 decode groups

A 2:1 pool ratio for this mix — and the point of writing it as two independent divisions is that the ratio moves when the workload does. Documents get longer: prefill pool grows, decode untouched. Sessions get chattier: the reverse. A shared fleet can’t express that; two pools can.

The hop has to be paid for, so price it. One 8K-token 70B request’s KV, from §4:

KV to ship = 2.62 GB
over NVLink        900 GB/s:  2.62 / 900   ≈  3 ms
over 400 Gb RDMA    50 GB/s:  2.62 / 50    ≈ 52 ms

(RDMA — remote direct memory access — is the NIC writing straight into the peer machine’s memory with no CPU copies on either side; it is how KV moves between nodes.) Against a TTFT already in the hundreds of milliseconds, 3-52 ms of transfer is affordable — and engines hide most of it by shipping layer by layer, overlapped with the prefill still producing later layers. The same arithmetic says when disaggregation is wrong: tiny models, short prompts, or a fleet too small to keep two pools independently busy — then the transfer plus the operational complexity buys nothing chunked prefill wasn’t already delivering. Below roughly node scale, share; at fleet scale, split.

flowchart LR
    U([Requests]) --> RT["Router<br/>KV-aware"]
    RT --> P1["Prefill pool<br/>compute-bound<br/>scales on prompt tok/s"]
    P1 -->|"KV over NVLink / RDMA"| D1["Decode pool<br/>bandwidth-bound<br/>scales on concurrent seqs"]
    RT -->|"prefix already warm here"| D1
    D1 --> O([Streamed tokens])

    style P1 fill:#2d6a4f,color:#fff
    style D1 fill:#bc6c25,color:#fff
    style RT fill:#40916c,color:#fff

KV-cache-aware routing

Once KV caches are first-class objects that live somewhere, where a request lands starts to matter. KV-cache-aware routing sends a request to the worker that already holds the KV blocks for its prefix, instead of load-balancing round-robin and re-prefilling from scratch.

The workload that makes this enormous is multi-turn chat, and the arithmetic is Deriving the numbers statelessness fact seen from the server side: turn t’s prompt is turn t−1’s entire conversation plus one message, so the prefix-reuse rate approaches 100%. Route turn t to the worker that served turn t−1 and the “prefill” is one message long; route it anywhere else and the whole conversation re-prefills. This is the serving-side mechanism underneath the prompt caching you already price at the API level — Prompt caching derived derived the billing rules from causal attention; down here, “cache hit” literally means “the K/V blocks are resident in that worker’s HBM, and §5’s prefix sharing points the new sequence at them without copying.”

Size the win on the repo’s own reference agent (chapter 09 runs it at 6,000 prefix tokens plus 1,200 per turn). At turn 20 the accumulated context is 6,000 + 19 × 1,200 = 28,800 tokens. Serve it on the 8B and compare a cold worker against the warm one:

cold worker: prefill 28,800 tokens = 2 × 8e9 × 28,800 = 461 TFLOP
             at 50% of peak        = 461e12 / 494e12  = 932 ms of TTFT
warm worker: prefill 1,200 new     = 2 × 8e9 × 1,200  = 19.2 TFLOP
             at 50% of peak        = 19.2e12 / 494e12 =  39 ms of TTFT

routing hit vs miss ≈ 24× on TTFT, and the gap widens every turn

A router that lands 90% of turns warm versus one that lands 30% is worth more TTFT than any kernel optimization in this chapter — which is why the routing layer, the least GPU-flavored component in the stack, has become a performance component.

The open example is the llm-d project — a Kubernetes-native inference scheduler whose router scores workers by prefix overlap before dispatching — and Dynamo ships a smart router doing the same job in the NVIDIA stack.

The failure mode to name out loud, because it is a distributed-systems problem wearing GPU clothes: prefix affinity fights load balancing. Pin every conversation to its warm worker and hot conversations hotspot one GPU while its neighbors idle; balance perfectly and every hop costs a full re-prefill. Real routers score both — overlap × current load — and accept that the KV cache has become a piece of distributed state with all the usual placement, eviction, and consistency headaches. If you have run a sticky-session tier or a sharded cache, you have debugged this exact tension before.

9. Measuring it — utilization, MFU, and the Nsight pair

Every claim so far was derived. On the job you have to verify them on a live system — and the first tool everyone reaches for, nvidia-smi, measures almost nothing.

Why “GPU util: 95%” is nearly meaningless

nvidia-smi’s utilization is kernel residency: the fraction of the sample window during which at least one kernel was executing. Not how many SMs that kernel used. Not whether it computed or waited on memory. One kernel, one SM, running continuously:

a kernel occupying 1 of 132 SMs, always resident   ->  "GPU util: 100%"
actual compute in use                              ≤ 1/132 ≈ 0.8%

And the trap is not hypothetical — it is this chapter’s headline case. Batch-1 decode keeps a kernel resident essentially always, so nvidia-smi reads ~100% while §2 proved the chip is doing 0.34% of its possible arithmetic. “Util is high, we need more GPUs” is exactly the wrong conclusion drawn from exactly this number — the fix was batch size, which is free.

The memory gauge misleads the same way for a different reason: serving engines preallocate the KV space (vLLM grabs ~90% of HBM at startup and manages it internally), so “memory used” reads pinned near full whether the cache holds one sequence or seven hundred. Both headline numbers, useless for the two questions you actually have. Use the engine’s own counters for occupancy of the KV pool, and the two ratios below for efficiency.

The two ratios that mean something

• MFU — model FLOPs utilization: the FLOPs implied by your token throughput, divided by peak. tokens/s × 2 × params / peak FLOP/s.
• MBU — model bandwidth utilization: the bytes your decode must stream per second (weights plus live KV), divided by peak bandwidth.

Which one matters is phase-dependent, and that is the punchline of the whole measurement story. Decode first — 8B, batch 1, a healthy deployment measuring 146 tok/s:

bytes per token   = 16e9 (weights; short context, KV negligible)
bytes/s           = 146 × 16e9        = 2.34e12
MBU               = 2.34e12 / 3.35e12 = 70%

FLOP/s            = 146 × 2 × 8e9     = 2.34e12
MFU               = 2.34e12 / 989e12  = 0.24%

Same measurement, two denominators, opposite verdicts: this decode is healthy — 70% of the physically possible bandwidth, closing on the 209 tok/s ceiling — while its MFU looks catastrophic and is irrelevant. And notice what MBU hands you that no counter does: a distance-to-ceiling. 70% achieved means the remaining headroom on this workload is at most 1/0.7 ≈ 1.4×, so anyone promising a 5× kernel win on this decode is proposing to break physics — the 5× has to come from a lever that changes the workload (batch, quantization), not the kernel. Now prefill, using Latency loaded-endpoint observation of ~20,000 prompt tok/s on the same class of model:

FLOP/s = 20,000 × 2 × 8e9 = 3.2e14
MFU    = 3.2e14 / 989e12  = 32%

A third of peak — decent, with headroom worth profiling for. Judge decode by MBU, judge prefill by MFU (40-60% is strong; the 50% assumed in §3 and §6 sits in that band). Quoting the wrong ratio for the phase is the measurement version of quoting sparse TFLOPS, and interviewers use it as a shibboleth.

What goes on the dashboard

Before the profilers — which are for incidents and tuning sessions, not for Tuesdays — the standing metrics that make either ratio computable when you need it:

• Tokens per second, prefill and decode separately. One combined number hides the phase split that every diagnosis starts from.
• TTFT p50/p90 and ITL p50/p99. The tails carry the diagnoses: p99 ITL spiking with long-prompt arrivals is §3’s head-of-line signature.
• MBU and MFU, computed from the throughput numbers above and the fixed model constants — two multiplications, no profiler needed.
• KV-pool occupancy and preemptions per minute, from the engine. Occupancy pinned at 100% with preemptions climbing means the batch is capacity-capped: the §4 wall, not the scheduler.
• Batch occupancy per iteration. If it never approaches its configured cap while the queue is nonempty, something upstream (admission, KV, token budget) is the binding constraint.

Nothing in that list requires Nsight; all of it decides whether you reach for Nsight and for which of the two.

The Nsight pair

Two profilers, two altitudes — and the division of labor matters more than any individual feature, because using them in the wrong order wastes days.

Nsight Systems (nsys profile -o trace <your serve command>) is the system-wide timeline: every kernel launch, memcpy, NCCL collective, and CPU thread on one time axis, viewed in a GUI afterwards. It answers “where did the wall-clock go?” — the questions between kernels:

• Gaps. Whitespace on the GPU row while the CPU row is busy means launch overhead is pacing the GPU — decode steps are thousands of microsecond-scale kernels, and the CPU can fail to keep up. The standard fix is CUDA Graphs: record the step’s whole launch sequence once, replay it as a single unit, and the gaps close. (Engines enable this by default; a config change can silently disable it, and the timeline is where you catch that.)
• Serialization. At TP, do NCCL collectives overlap compute, or does the timeline alternate compute-silence-compute? The §7 overhead arithmetic assumed overlap; here is where you check it.
• Foreign work on the critical path. Host-to-device copies before each step, tokenization on the serving thread, a Python scheduler stall — all visible as CPU-row activity that the GPU row waits on.

Nsight Compute (ncu) is the per-kernel microscope: it replays one kernel (much slower — you sample, never wrap the whole server) and reports why that kernel is slow. It answers “is this kernel at its roofline?”:

• The Speed of Light section: this kernel’s achieved percentage of peak compute and of peak memory throughput, side by side — the roofline verdict, per kernel. A decode matrix-vector kernel showing memory ~80%, compute ~5% is §2 confirmed empirically; the same shape on a prefill GEMM is a bug.
• Occupancy — resident warps per SM as a fraction of the maximum the SM can host, i.e. how much latency-hiding parallelism (§1) the kernel offers the scheduler. Low occupancy plus low memory throughput means the kernel can’t even keep the memory system busy — too few threads in flight to cover latency.
• Achieved bandwidth in GB/s, to lay directly against the 3.35 TB/s spec.

The workflow is always the same order: nsys to find which kernel or gap owns the time, then ncu on that kernel to find why. Profiling kernel-by-kernel before looking at the timeline is how people spend a week optimizing a kernel that was 4% of the step.

What a healthy decode profile looks like

The checklist form, for reading someone else’s trace or your own:

• Timeline dense: kernel gaps under a few percent of step time (CUDA Graphs on and actually engaged).
• The step dominated by a few fused kernels — attention and the big matrix-vector multiplies — not thousands of slivers.
• Those kernels’ Speed of Light: memory high (60-80%+), compute low. That is correct for decode — do not “fix” it.
• Aggregate MBU at 60-80%; if it reads 20%, suspect batch too small, kernels unfused, or KV scattered pathologically.
• TP only: NCCL time overlapped with compute or under ~10% of the step.
• Batch pinned at the KV-memory or ITL limit, not at a default someone never raised.

Honesty clause: this section gives you the reading skill — what the timeline, the SOL panel, and the two ratios mean, and which conclusion follows from which. Fluency with the tools themselves comes only from opening them on a real serve, which is the second half of §5’s weekend project: wrap the vLLM sweep in nsys profile, find the decode step in the GUI, and check the checklist against it once. One rep is enough to talk about it concretely; zero reps is audible.

10. The levers, ranked

Everything above, folded into the table you actually use — symptom first, because that is how the problem arrives:

Two rows deserve their arithmetic, because both are pure applications of §2.

Quantization stores weights in fewer bytes — INT8 or FP8, one byte per parameter — while computing in higher precision. Halve the bytes and decode intensity doubles, which on the memory roof means the ceiling doubles:

8B in INT8:  weights = 8e9 × 1 B                = 8 GB
             time per token = 8e9 / 3.35e12     = 2.39 ms
             decode ceiling = 1 / 0.00239       = 419 tok/s     (was 209)
             intensity = 2 FLOPs / 1 byte       = 2 FLOP per byte

Weights-only quantization is the safe first step (the KV cache and activations are separate, harder decisions), and the honest caveat is that it is the first lever in this table that can touch output quality — measure on your evals before believing the free 2×.

Speculative decoding spends the FLOPs decode leaves idle. A small draft model proposes k tokens cheaply; the target model verifies all k in one forward pass — and because that pass is memory-bound, verifying k+1 positions streams the same 16 GB of weights as generating one. With acceptance rate α per token, the expected tokens banked per target pass:

expected tokens = 1 + α + α² + ... + α^k

k = 4, α = 0.7:  1 + 0.7 + 0.49 + 0.343 + 0.240  = 2.77 tokens per pass
                 -> up to ~2.8× lower single-stream latency

The trade is explicit: total compute goes up (a whole draft model runs, and rejected tokens are wasted work) to buy latency down — precisely the right trade on a machine whose §2 diagnosis is 99.7% idle arithmetic, and precisely the wrong one on a compute-saturated prefill pool.

Order of operations mirrors chapter 09’s discipline: measure first (MBU/MFU, the ITL distribution, KV-pool usage and preemptions — no denominator, no ratio); then the free structural wins (paged KV and continuous batching, which a modern engine gives you by default — confirm, don’t assume); then scheduling policy (chunked prefill, token budgets); and only then the levers that cost quality, hardware, or complexity (quantization, parallelism changes, disaggregation). Each step changes the numbers feeding the next, so the order is forced, not stylistic.

The mechanism → lever map

The chapter in one table, in chapter 00’s closing format: cover the right column and regenerate it from the left.

Cheat sheet

Numbers are representative (H100 SXM, BF16, spec-sheet peaks); re-derive on your SKU before quoting any of them in anger.

What interviewers probe

“Why is decode memory-bound? Derive it.” Say: generating one token multiplies through every parameter — about 2 FLOPs per parameter — and at batch 1 every 2-byte parameter must stream from HBM, because 16-plus gigabytes of weights can’t live in 50 MB of SRAM. That’s 1 FLOP per byte, against a ridge point of 989 TFLOPS over 3.35 TB/s ≈ 295 FLOP per byte. So attainable throughput is 1 × 3.35 TB/s = 3.35 TFLOP/s — a third of a percent of peak. The fix is intensity: batch B sequences and each weight-read serves B tokens — with the caveat that KV reads don’t amortize, so long contexts drag effective intensity back down.

“Size the KV cache for a 70B model at 8K context.” Say: 2 for K and V, times 80 layers, times 8 KV heads — it’s GQA; that 8 would be 64 under vanilla multi-head, an 8× difference, so always check — times head dim 128, times 2 bytes for fp16: 320 KB per token, 2.62 GB per 8K sequence. Then land the capacity punch: the weights alone are 140 GB against an 80 GB card, so this model is tensor-parallel before the cache stores its first token, and at TP=4 the aggregate free memory holds about sixty 8K conversations.

“nvidia-smi shows 95% GPU util but throughput is bad. What’s wrong?” Say: that metric is kernel residency — any kernel running during the sample window counts the whole window — so a batch-1 decode reads ~100% while doing 0.3% of possible FLOPs. Measure MBU for decode and MFU for prefill instead. If MBU is high, the GPU is honestly at its memory roof and the lever is batch or quantization; if MBU is also low, profile — nsys for launch gaps and serialization on the timeline, then ncu Speed-of-Light on the dominant kernel. Most likely root cause: batch too small, usually capped by KV memory.

“When tensor parallelism vs pipeline parallelism?” Say: TP splits every matrix and pays two all-reduces per layer — 160 latency-bound collectives per token on an 80-layer model — so it needs NVLink and stays inside a node; in exchange it divides per-token latency by p and is the only way a 140 GB model exists at all. PP splits the layer stack, ships one small activation per stage boundary, crosses nodes happily, but pays the bubble — (p−1)/(m+p−1) — so it needs microbatch traffic and never helps a single stream. Standard layout: TP to the edge of the NVLink domain, PP across nodes on top; a 405B in BF16 is 810 GB, which is TP8 × PP2 across two nodes at ~51 GB per GPU.

“Your P99 ITL is terrible but the mean is fine.” Say: classic head-of-line blocking — long prefills freezing in-flight decodes; an 8K prompt is ~260 ms of solid compute walking in front of a 6 ms heartbeat, so p99 spikes exactly when long prompts arrive while the mean barely moves. Confirm by correlating ITL spikes with prompt-arrival sizes, then enable chunked prefill and tune the per-iteration token budget, accepting a bounded TTFT cost. If the fleet is large and the two SLOs keep fighting, that’s the case for disaggregating prefill and decode pools and optimizing goodput rather than tokens per second.

“How would you raise throughput without buying GPUs?” Say: in order — confirm the engine pages its KV and batches continuously, because that’s the 2-4× class of win and a misconfiguration can silently forfeit it; raise concurrency until KV memory or the ITL target caps it; quantize weights to FP8/INT8, which halves bytes per parameter and doubles the decode ceiling; and if traffic is multi-turn, add prefix caching with KV-aware routing so conversations stop re-prefilling. Every step is the same roofline argument: more FLOPs per HBM byte, or fewer bytes per token.

“What batch size should we run?” Say: it’s not a free parameter — it’s pinned between two constraints you can compute. The ceiling is the ITL SLO, because every added sequence adds its KV stream to the step (§5’s arithmetic showed batch 57 → 760 taking the step from 6 to 23 ms). The cap is KV memory: free HBM over per-sequence cache, so for the 8B at worst-case 8K contexts, 60 GB over 1.05 GB ≈ 57 — which is why paged KV, by shrinking the effective per-sequence footprint to actual usage, is what raises the cap. Then sweep between the two on real traffic and pick the largest batch that holds p99 ITL; the knee in that curve is the KV pool saturating.

“Prefill is supposed to be compute-bound — why is your prefill MFU only 30%?” Say: MFU counts only the useful model FLOPs, and real prefill spends time on things that aren’t the big matmuls — the quadratic attention kernels, normalizations, kernel-launch overhead between them — plus scheduling effects like chunked prefill deliberately splitting the work and short prompts that never amortize the weight-read (a prompt under ~295 tokens is still memory-bound by the intensity formula). 30-50% is the normal band; the way to find the specific gap is ncu on the dominant kernels — if the big GEMMs individually show high compute Speed-of-Light, the loss is between kernels, not in them.

“You keep saying roofline — what’s on the axes?” Say: x is arithmetic intensity — FLOPs performed per byte moved from HBM; y is attainable FLOP/s — the min of peak compute and intensity times bandwidth. The ridge sits at ~295 FLOP/byte on an H100 with dense BF16 peaks. Decode at batch 1 sits at x ≈ 1, prefill at x ≈ prompt length; batching, quantization, and speculative decoding are all just moves on that one plot. Then offer to draw it — it’s the diagram the whole conversation stands on.