LLM Inference Performance

In this lesson, we’ll trace how most serving decisions (batch size, scheduler, parallelism layout, which GPU to buy) follow from one ratio: how much arithmetic the chip can do per byte it pulls from its own memory. The KV cache explains why decode is memory-bound, and prefill vs decode prices the consequence at the API level. We’ll work at the hardware level, where the same fact becomes a number you can compute, then cover the serving stack built on it: PagedAttention, continuous batching, chunked prefill, tensor and pipeline parallelism, disaggregated serving, and the profilers that tell you which one you need.

The discipline is the same one you use when a service is capped by disk reads instead of CPU: the hardware differs, the reasoning is identical. Every number below is reproducible with a calculator.

By the end you’ll be able to:

• Place prefill and decode on a roofline and see why batch-1 decode runs at ~0.3% of the GPU’s peak.
• Size a KV cache from four model dimensions and convert free memory into a max-concurrency number.
• Explain what PagedAttention borrows from OS virtual memory and what it buys.
• Choose continuous vs chunked prefill, and tensor vs pipeline parallelism, from the arithmetic.
• Read the Nsight profilers and explain why nvidia-smi’s “GPU util” is misleading.

Assumptions, fixed once. Unless a line says otherwise, every number below assumes an NVIDIA H100 SXM: 80 GB of HBM at 3.35 TB/s, 989 TFLOPS of dense BF16 compute, 900 GB/s of NVLink, 64 GB/s of PCIe Gen5 x16. These are representative spec-sheet values; real SKUs vary, and none of the reasoning changes when they do. BF16 means 2 bytes per number. Two reference model shapes recur: an 8B (32 layers, hidden width 4,096, 8 KV heads, head dimension 128) and a 70B (80 layers, hidden width 8,192, 8 KV heads, head dimension 128), both Llama-3-shaped so the arithmetic checks against public configs.

The memory hierarchy, and why serving is a bandwidth problem

A GPU is not a faster CPU. It is built on one assumption: that the problem is thousands of identical operations wide, so it trades single-thread speed for tens of thousands of simple arithmetic units running in lockstep.

Two terms first. A kernel is one function launched onto the GPU (“multiply these two matrices,” “apply softmax to these rows”) run by thousands of threads at once; a serving engine’s decode step is a stream of kernel launches. An SM (streaming multiprocessor) is the unit those threads run on: the H100 SXM has 132, each holding arithmetic units, a slice of fast on-chip memory, and a scheduler.

A CPU hides memory latency with large caches; a GPU hides it with more parallelism. Each SM keeps many groups of threads resident, and when one stalls on a memory read, the scheduler runs another. This works only while there is enough independent work in flight, and batch-1 decode is exactly the workload that fails to supply it.

The pyramid

Arithmetic units are useless if the data is not next to them. The memory system trades capacity for speed at each level:

SRAM is on the die itself: the only memory fast enough to keep the arithmetic units fed, and four orders of magnitude too small to hold a model. HBM is the DRAM stack on the GPU package; its 80 GB is the capacity in every “does it fit” question, and its 3.35 TB/s is the bandwidth this lesson centers on. NVLink is the in-server GPU-to-GPU interconnect, 3.7× slower than HBM but 14× faster than PCIe, the general-purpose bus to the host. That 14× gap decides where multi-GPU schemes can live.

flowchart TD
    R["Registers<br/>KBs per SM, ~1 ns"] -->|"within the SM"| S["SRAM: shared mem + L2<br/>~50 MB L2, tens of TB/s"]
    S -->|"on-die"| H["HBM<br/>80 GB @ 3.35 TB/s"]
    H -->|"NVLink ~900 GB/s"| G["Other GPUs in the node"]
    H -->|"PCIe ~64 GB/s"| D["Host DRAM / NVMe"]

    style H fill:#2d6a4f,color:#fff
    style G fill:#40916c,color:#fff
    style D fill:#bc6c25,color:#fff

The ratio that runs the lesson

A FLOP is one floating-point operation. Divide the two headline numbers:

peak compute / HBM bandwidth = 989e12 / 3.35e12 = 295 FLOPs per byte

For every byte the chip fetches from HBM, it has time to do ~295 operations before the memory system becomes the reason it waits. Feed it fewer than 295 FLOPs of work per byte and the arithmetic units idle: the workload is memory-bound, throughput set by bytes moved. Feed it more and it is compute-bound, the arithmetic units are the wall. This is the same shape as a service where every request scans data from disk: past a point, adding CPU does nothing. On a GPU serving decode, the “disk” is HBM.

Two consequences before the roofline formalizes them. First, a floor you can compute in your head: touching everything in HBM once (which a decode step roughly does) takes 80 GB / 3.35 TB/s ≈ 24 ms, so any step that must sweep the whole card is capped at ~40 steps/s regardless of the arithmetic units. Second, FlashAttention’s main contribution is a direct application of the ratio: it tiles attention so scores are produced, used, and discarded inside SRAM, spending extra FLOPs to avoid writing the score matrix to HBM. On a chip with 295 FLOPs to spare per byte, that is a good trade.

Almost every term in an inference job (batching, paged KV, quantization, speculative decoding, disaggregation) is a scheme for changing the FLOPs-per-byte of decode, or for spending the FLOPs decode leaves idle.

Roofline modeling

The roofline tells you, for any kernel, what throughput it is allowed to reach on a chip before you profile anything. Arithmetic intensity is the FLOPs a workload performs per byte it moves from HBM. The ceiling has two segments:

attainable FLOP/s = min( peak compute , intensity × bandwidth )

At low intensity the second term wins: you pay for bytes, and doubling intensity doubles throughput, a rising slope whose gradient is memory bandwidth. At high intensity the flat compute roof caps you. The corner where they meet is the ridge point, 989e12 / 3.35e12 = 295 FLOP/byte.

One note on the numerator, the most common way to be wrong by 2×. NVIDIA’s headline 1,979 TFLOPS BF16 for the H100 assumes 2:4 structured sparsity (two of every four weights zero and skipped). Served LLM weights are dense; the dense peak is 989. Quoting the sparse figure halves your ridge point and doubles your apparent efficiency.

 attainable                schematic, log-log axes, shape only
 FLOP/s
   989 T |. . . . . . . . . . . _______________●________   compute roof
         |                     /                prefill,
         |         memory     /                 2,048-token prompt
         |         roof:     /
         |         slope =  /
  3.35 T |. . ●. . . . . . /
         |  decode,       /  ridge = 295
         |  batch 1      /
         +----1--------- 295 -------- 2,048 --------------
                    arithmetic intensity, FLOP per byte

Placing decode on it

Generating one token multiplies activations through essentially every parameter. A matrix multiply does one multiply and one add per parameter, so a forward pass costs ~2 FLOPs per parameter per token. Each BF16 parameter is 2 bytes, and at batch 1 every one must be streamed from HBM for every token. 16 GB of weights cannot live in 50 MB of SRAM, so nothing carries over.

decode intensity (batch 1) = 2 FLOPs per param / 2 bytes per param = 1 FLOP per byte

Models are served in 16-bit because on the memory roof, bytes are throughput: FP32 would double every byte count here and halve every decode ceiling for accuracy nothing downstream can measure. (Quantization, later, continues that logic past 16.)

One FLOP/byte against a ridge of 295: batch-1 decode sits at the far left, and the roofline gives the cost: min(989e12, 1 × 3.35e12) = 3.35e12, which is 0.34% of peak. Not misconfiguration: the workload cannot feed the arithmetic units. In the time domain on the 8B model, 16 GB of weights over 3.35 TB/s is ~4.8 ms/token, a decode ceiling of ~209 tok/s, while the arithmetic itself (16 GFLOP over 989 TFLOPS) takes only ~16 µs. The chip waits ~295× longer than it computes, the ridge point again, measured in time.

Placing prefill on it

Prefill processes the whole prompt in one pass, so each fetched weight is used against every prompt token before being discarded:

prefill intensity ≈ n_tokens FLOP per byte
2,048-token prompt: ≈ 2,048 >> ridge 295 -> compute-bound

Same model, same GPU, opposite side of the ridge. An 8B prefill of 2,048 tokens is ~33 TFLOP: ~33 ms at the full roof, ~66 ms at a realistic 50% of peak (attention, normalization, and launch overhead keep no real workload on the roof; 40–60% is a strong prefill). One boundary case falls out of the formula: intensity ≈ prompt length, so a prompt shorter than ~295 tokens is still memory-bound. “Prefill is compute-bound” holds only for prompts long enough to amortize the weight-read.

Batching moves decode up the roofline

Batch-1 decode has one exit: make each weight-read serve more than one token. Decode B sequences together and each streamed parameter contributes to B generations, so decode intensity ≈ B FLOP/byte. Break-even with the ridge needs B ≈ 295. Decode wants a batch in the hundreds before the compute roof is even visible.

Two caveats. First, the KV cache does not amortize. Weights are shared across the batch; each sequence’s cache is its own and is streamed in full every step. With both terms:

effective intensity = B × 2P / ( 2P + B × kv_bytes )
8B, B=64, short chats (~600 live tokens): intensity ≈ 49  (not 64)
8B, B=64, long contexts (8,000 tokens):   intensity ≈ 12  (not 64)

Same batch, a quarter of the intensity, purely because the caches grew. Long-context serving is memory-bound twice over, in capacity and bandwidth. Second, holding hundreds of sequences resident is a capacity problem: every one needs its KV cache in HBM at once, which is why the memory manager below is one of the largest throughput features ever shipped for serving.

For any performance question, draw the roofline first: ridge at ~295, decode at 1, prefill at prompt-length. “Should we quantize?” is “move decode right by shrinking bytes.” “Why is util high but throughput low?” is “you are on the memory roof, and util doesn’t measure that.”

Prefill and decode place different demands

The user-visible metrics split along the phase boundary:

• TTFT, time to first token: queueing plus the entire prefill. The delay before anything appears.
• ITL, inter-token latency: the gap between one streamed token and the next. Its spikes are what users perceive as stutter.
• TPOT, time per output token: the mean of the ITLs. TPOT summarizes; ITL’s distribution diagnoses.

(Latency budgets a whole request in these terms from the client side; here the concern is what one GPU does to them when it serves both phases at once.)

Put numbers on both, 8B model. A 2,048-token prompt prefills in ~66 ms, so TTFT is well under typical targets before queueing. Decode at a healthy batch runs a step every ~6 ms (derived below), so a smooth stream shows ITL ≈ 6 ms. For a 250-token response that is 66 ms + 250 × 6 ms ≈ 1.57 s total, of which decode is ~96%, the phase running at 0.3% of the chip’s arithmetic. That is why the latency levers that matter target decode: shortening the prompt only attacks the other 4%.

Head-of-line blocking

A serving GPU never runs one request. It holds a batch of in-flight decodes, each wanting a small regular ~6 ms step, and a queue of new arrivals, each wanting a prefill that is a large block of compute. They share one set of SMs. When a long prompt arrives and a naive scheduler runs it whole:

8,000-token prompt prefill = 2 × 8e9 × 8,000 = 128 TFLOP
at 50% of peak = 128e12 / 494e12 ≈ 259 ms
-> every in-flight decode stalls ~260 ms  (worst ITL ≈ 43× the 6 ms baseline)

One long prompt turns every concurrent user’s smooth stream into a quarter-second freeze. That is head-of-line blocking, and it is why ITL’s distribution is the useful signal: the mean barely moves while p99 spikes with long-prompt arrivals.

Priorities alone cannot fix it. “Decode always preempts prefill” starves TTFT; “prefill first” is the 260 ms freeze as policy. The phases want opposite things from the same hardware:

The bottom row also says the two phases do not autoscale on the same signal. This tension resurfaces twice: chunked prefill makes them share one GPU without interference (a scheduling answer), and disaggregation stops making them share at all (an architecture answer). Which you need is a fleet-size question.

The KV cache, sized honestly

On a serving GPU the KV cache decides your batch size, and batch size is the primary lever. Attention stores one K and one V vector per token, in every layer, per KV head:

bytes per token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

n_kv_heads is its own factor because in GQA (grouped-query attention) several query heads share one K/V pair, precisely so this number is smaller than the query-head count. MQA (multi-query attention) is the limit case: one shared K/V pair for all query heads. Both are training-time decisions whose purpose is visible in this serving-time formula. (Attention and why context costs what it does is the mechanism.)

The 70B shape (80 layers, 8 KV heads via GQA, head_dim 128, FP16) gives 2 × 80 × 8 × 128 × 2 = 320 KB per token, so an 8,000-token sequence needs 2.62 GB. The formula is linear in context:

A single 128K-context conversation uses more than half an H100’s memory in cache alone. Long-context serving is a regime where a handful of requests exhaust the card.

Why 70B needs tensor parallelism first

70e9 × 2 B = 140 GB of weights against an 80 GB card: the weights alone are 1.75× the card, so a 70B model in 16-bit cannot fit on one H100 before any cache or batch. This is about fit, not speed, and is the first reason 70B-class serving is tensor-parallel. At TP=4 the weights shard to 35 GB per GPU and the KV cache shards with the heads, leaving ~41 GB per GPU for KV, ~164 GB aggregate, ≈ 62 concurrent 8K sequences.

Max concurrency on the model that does fit

The 8B on one H100: 2 × 32 × 8 × 128 × 2 = 128 KB/token, so an 8K sequence is 1.05 GB. With 16 GB of weights and ~4 GB reserved for runtime and activations, 80 − 20 = 60 GB is free for KV, giving 60 / 1.05 ≈ 57 concurrent worst-case sequences. But decode wants a batch near 295 to reach the ridge. Even with memory fully packed, decode stays memory-bound. The batching lever runs out of memory before it runs out of usefulness. So every byte shaved off the cache converts directly into batch, and batch into throughput.

That is why KV-shrinking is a model-design choice, made before a serving engineer sees the model:

GQA is an 8× cache reduction taken before any serving technique runs. Quantizing the KV cache to FP8 halves dtype_bytes for another 2×, and the two stack.

Where the two decode ceilings meet

Two ceilings are now in circulation and look inconsistent: ~209 tok/s counting only the weights, and (from the KV cache) ~102 tok/s counting only the cache at 100K context. Neither is wrong; a real step pays both streams, (weight bytes + live KV bytes) / bandwidth. For the 8B at batch 1: ~208 tok/s at 1K tokens (weights dominate), ~115 tok/s at 100K tokens (cache dominates). The crossover, where one conversation’s cache costs as much bandwidth as the entire model, is 16e9 / 131,072 ≈ 122,000 tokens. Per-sequence decode cost is not flat; it degrades with context length even after the memory to hold it is paid, which is the hardware reason long conversations get slower and why context discipline is a performance feature, not only a cost one.

The trap in sizing a KV cache is the head count: using 64 heads for a GQA model inflates the answer 8×. Name the formula, state whether the model uses GQA, and substitute explicitly.

PagedAttention: virtual memory for the KV cache

The sizing above assumed every allocated KV byte holds a real token. Before 2023, most did not, and fixing that produced the largest single throughput win in the modern stack.

The fragmentation problem

A sequence’s cache grows one token at a time toward a length nobody knows in advance. The attention kernels of the day required each cache to be contiguous, and the only safe contiguous allocation for an unknown length is the worst case: reserve max_len up front. An 8B at max_len 8,192 reserves 1.07 GB per request; a chat turn using 300 tokens needs 39 MB, ~96% internal fragmentation, memory allocated to a request and holding nothing. Pre-paged systems wasted KV memory three ways:

• Internal fragmentation: the worst-case reservation, by far the biggest term.
• External fragmentation: variable-sized slabs leave unusable gaps, like a heap with no compactor.
• Duplication: sequences sharing a prefix each stored their own copy of its K/V.

Across realistic traffic the vLLM paper measured 60–80% of KV memory wasted. The GPU was not short of memory but short of usable memory, and the primary lever was capped by the waste.

The fix, borrowed from operating systems

PagedAttention applies OS virtual memory to the KV cache:

• Carve KV memory into fixed-size blocks of, say, 16 tokens (the page).
• Give each sequence a block table mapping its logical positions to whichever physical blocks it owns, no contiguity required (the page table).
• Allocate a new block only when the previous one fills, on demand, not on arrival.

The attention kernel walks the block table the way a CPU walks page tables. Fragmentation collapses to at most one partially-filled block per sequence: 16 tokens × 131,072 B ≈ 2 MB, versus the ~1 GB reservation. The block size is the same trade-off page size is in an OS: smaller blocks bound waste tighter but scatter reads and lengthen tables; bigger blocks read contiguously but let internal fragmentation return. Implementations landed around 16 tokens.

Two more OS ideas eliminate the duplication term. Sequences sharing a prefix point their block tables at the same physical blocks (prefix sharing, one copy of the system prompt however many users share it), and a shared block is copied only on write (copy-on-write, fork semantics). A 2,000-token system prompt at 100 users drops from 26.2 GB of identical bytes to 0.26 GB.

When memory still runs out, the engine preempts: it evicts a sequence’s blocks and later recomputes them or restores them from host DRAM over PCIe. A nonzero preemption rate signals that the KV budget, not the scheduler, is the binding constraint.

What it buys, in tokens per second

The point was batch, not memory hygiene. Same 8B card, 60 GB of KV space, a mixed workload averaging 600 live tokens per sequence (78.6 MB each). Naive reservation holds ~57 sequences; paged allocation holds ~760. The subtlety is that wasted space is not wasted traffic. The naive system never reads its empty reservations, so both stream only live bytes. What changes is how many sequences share each pass over the weights:

naive:  16 + 57 × 0.0786  = 20.5 GB/step -> 6.1 ms  ->  ~9,300 tok/s
paged:  16 + 760 × 0.0786 = 75.7 GB/step -> 22.6 ms ->  ~33,600 tok/s

~3.6× the throughput from the same hardware (inside the 2–4× the vLLM paper reported), by the same mechanism as batching: more sequences amortizing each weight-read. Note the trade the arithmetic exposes: ITL rose from 6.1 to 22.6 ms, because a bigger batch makes every step heavier. Throughput and per-token latency trade against each other, so production schedulers cap batch to hold an ITL target instead of maximizing tokens/s.

vLLM is the reference implementation and default open-source stack; PagedAttention has been adopted almost everywhere (TensorRT-LLM, SGLang) to the point that “the KV cache is paged” is now an assumption, not a feature. To see it, serve an 8B with vLLM, sweep --max-num-seqs, and plot tokens/s and p99 ITL against batch while watching the KV-usage and preemption counters: throughput climbs near-linearly, then flattens as KV memory saturates, ITL rises throughout, and preemptions appear where the curve bends.

Batching strategies

Batching is the lever; the strategy question is when requests may join and leave the batch. Three generations, each fixing the previous one’s idle time.

Static batching, inherited from the translation era, collects B requests, pads them to equal length, runs the batch to completion, and admits nobody until it drains. It assumes uniform output lengths; LLM generation lengths vary by two orders of magnitude and are unknown in advance. A batch of 8 with outputs of 100…800 tokens runs 6,400 slot-steps to do 3,600 useful ones (56% utilization) and one 800-token response blocks new admission for seconds.

Continuous batching (the Orca paper’s iteration-level scheduling) is the fix. Decode has a natural preemption point every few milliseconds, the token boundary, where every sequence does the same thing: one forward pass, one token. So schedule iterations instead of requests: after each step, finished sequences exit and waiting requests join. A slot that frees at token 100 serves a new user at token 101.

flowchart LR
    subgraph ST["Static - slots wait for the longest request"]
        S1["step 1<br/>A B C D"] --> S2["step 150<br/>A · C ·"] --> S3["step 700<br/>A · · ·"] --> S4["drain, then<br/>admit E F G H"]
    end
    subgraph CB["Continuous - slots refill at token boundaries"]
        C1["step 1<br/>A B C D"] --> C2["step 150<br/>A E C F"] --> C3["step 700<br/>G E H F"] --> C4["never drains"]
    end

    style S2 fill:#9d0208,color:#fff
    style S3 fill:#9d0208,color:#fff
    style C2 fill:#2d6a4f,color:#fff
    style C3 fill:#2d6a4f,color:#fff

Utilization no longer depends on length variance, and a new arrival waits at most one iteration (milliseconds) instead of a batch-drain (seconds). This and paged KV ship together: join-anytime scheduling needs allocate-anytime memory, since a mid-flight admission cannot pre-reserve a contiguous gigabyte. Every modern engine runs the pair by default.

Chunked prefill (the Sarathi lineage) fixes what continuous batching still gets wrong: joining the batch requires a prefill, and a long one freezes every in-flight decode. It splits the prompt into fixed-size chunks (say 512 tokens) and co-schedules one chunk per iteration alongside the decode step:

8,000-token prompt, 512-token chunks = 16 chunks
iteration = decode 6.1 ms + chunk ~16.6 ms ≈ 23 ms
worst ITL during a long admission: 260 ms -> ~23 ms  (11× better)
TTFT for the long prompt: 16 × 23 ≈ 370 ms  vs  ~260 ms dedicated  (+40%)

That is the knob and its trade: chunk size buys ITL smoothness for the batch at a TTFT cost to the long prompt. Small chunks favor streaming chat; big chunks favor long-document TTFT. In configs it usually appears as a per-iteration token budget capping decode plus prefill-chunk tokens. Chunked prefill deliberately lowers prefill efficiency (each chunk re-reads weights) to protect a latency SLO, the standard soft-real-time trade, smooth over fast. When no chunk size satisfies both sides, that is the case for disaggregation.

Tensor and pipeline parallelism

140 GB of 70B weights cannot fit on an 80 GB card, so multi-GPU inference is the entry requirement, not an optimization. First set aside the distributed-systems reflex: “just add replicas” (the same full model behind a load balancer) scales request throughput once the model fits, but does nothing for a model that does not fit and nothing for per-token latency, because each replica still faces the memory wall alone. Tensor and pipeline parallelism split one copy of the model itself, which is why they cost communication.

Tensor parallelism: cut every matrix

Tensor parallelism (TP) splits each weight matrix across p GPUs (attention heads and MLP columns distributed), so every GPU computes a slice of every layer for every token. Partial results must be recombined twice per layer: the Megatron-style layout splits paired matrices column-then-row so the intermediate never travels, but each pair’s output is a partial sum every GPU needs in full. Two pairs per layer (attention output projection, MLP down-projection) means two all-reduces per layer per token. (An all-reduce is the collective where every GPU ends up holding the sum of all contributions; the library is NCCL, whose kernels appear by name in profiles.)

The bytes are trivial (one activation vector per collective, ~16 KB), but a 70B at TP=4 issues 2 × 80 = 160 synchronizing collectives per token, each with a fixed few-microsecond launch-plus-latency floor regardless of payload. Count latency, not bytes: 160 × ~5 µs ≈ 0.8 ms per token, ~8% on top of a 10.4 ms decode step over NVLink. The same 160 collectives over PCIe (~25 µs each) cost ~4 ms, ~40% overhead. That is the placement rule: TP communicates heavily per token, so it stays on NVLink, inside one node.

In exchange, TP is the only scheme that reduces per-token latency: each GPU streams 1/p of the weights, so the memory wall divides by p. A 70B at TP=4, batch 1, is 35 GB / 3.35 TB/s + 0.8 ms comm ≈ 11.2 ms, ~90 tok/s, for a model that could not fit on one card at all.

Pipeline parallelism: cut the layer stack

Pipeline parallelism (PP) gives each GPU a contiguous run of layers (a stage) and flows tokens through the stages: 80 layers at p=4 is 20 per stage. Communication is one 16 KB activation handoff per boundary, so the interconnect barely matters. PP is what crosses nodes.

The price is the pipeline bubble: while stage 1 works, later stages idle, and vice versa on drain. The fix is to split the batch into m microbatches fed back-to-back so stages overlap on different waves:

bubble fraction = (p − 1) / (m + p − 1)
p=4, m=16: 15.8% idle    p=4, m=64: 4.5%

The bubble shrinks only when microbatches greatly outnumber stages, which requires traffic. PP is a throughput regime, not a latency one: a token still visits all layers in sequence, so single-stream latency does not improve, and a deep pipeline under low traffic is mostly bubble.

Choosing, and combining

The standard layout follows: TP as far as the NVLink domain reaches (typically 8 GPUs in a node), PP across nodes only when a node is still too small. A 405B model in BF16 is 810 GB: TP=8 alone is 101 GB/GPU (doesn’t fit), TP=8 × PP=2 across two nodes is ~51 GB/GPU (fits, with room for KV).

flowchart LR
    subgraph N1["Node 1 — PP stage 1 (layers 1-40)"]
        A["GPU 0"] --- B["GPU 1"] --- C["... GPU 7"]
    end
    subgraph N2["Node 2 — PP stage 2 (layers 41-80)"]
        D["GPU 8"] --- E["GPU 9"] --- F["... GPU 15"]
    end
    N1 -->|"activation handoff (PP, inter-node)"| N2

    style N1 fill:#2d6a4f,color:#fff
    style N2 fill:#40916c,color:#fff

Within each node the 8 GPUs run TP over NVLink; between nodes a single activation crosses per token via PP. Training uses the same two words plus data parallelism and optimizer sharding, but optimizes step time over a corpus with gradients to synchronize, where inference only has to fit weights plus KV and hit a latency target. Scale covers that side.

Disaggregated serving and KV-aware routing

Chunked prefill made the two phases share a GPU without interference. The stronger move, once the fleet is large enough, is to stop making them share.

Disaggregated serving runs prefill and decode on separate GPU pools. A request lands on the prefill pool, which computes the TTFT-critical work as one uninterrupted compute-bound block; the KV cache it produces ships to a decode-pool worker, which streams tokens from it with a step nothing interrupts. The idea comes from the DistServe and Splitwise papers; Dynamo is NVIDIA’s productized version. Because the phases scale on different axes, separating them lets each pool be sized and scaled against its own bottleneck:

• The prefill pool is compute-bound: size it in prompt-tokens/s, scale it when TTFT degrades.
• The decode pool is bandwidth- and memory-bound: size it in concurrent-sequences-at-target-ITL, scale it when ITL degrades.

Interference is gone by construction. There is no long prompt to stall a decode, because they are on different machines. DistServe frames the win as goodput: requests/s that meet both the TTFT and ITL SLOs, instead of raw tokens/s, and shows shared GPUs sacrificing goodput even at high utilization. Sizing is two independent divisions: at 100 req/s, mean prompt 2,000 and output 250 tokens, with representative per-group capacities of ~10,000 prefill and ~2,500 decode tok/s, you get 20 prefill groups and 10 decode groups, a 2:1 ratio that moves when the workload does (longer documents grow only prefill; chattier sessions grow only decode). A shared fleet cannot express that.

The hop has a cost. One 8K-token 70B request’s KV is 2.62 GB: ~3 ms over NVLink, ~52 ms over 400 Gb RDMA (remote direct memory access, the NIC writing straight into the peer’s memory with no CPU copies, how KV moves between nodes). Against a TTFT already in the hundreds of milliseconds this is affordable, and engines hide most of it by shipping layer by layer, overlapped with the prefill still producing later layers. The same arithmetic says when disaggregation is wrong: small models, short prompts, or a fleet too small to keep two pools busy, where the transfer and operational complexity buy nothing chunked prefill was not already delivering. Below roughly node scale, share; at fleet scale, split.

flowchart LR
    U([Requests]) --> RT["Router<br/>KV-aware"]
    RT --> P1["Prefill pool<br/>compute-bound<br/>scales on prompt tok/s"]
    P1 -->|"KV over NVLink / RDMA"| D1["Decode pool<br/>bandwidth-bound<br/>scales on concurrent seqs"]
    RT -->|"prefix already warm here"| D1
    D1 --> O([Streamed tokens])

    style P1 fill:#2d6a4f,color:#fff
    style D1 fill:#bc6c25,color:#fff
    style RT fill:#40916c,color:#fff

KV-cache-aware routing

Once KV caches live somewhere, where a request lands matters. KV-cache-aware routing sends a request to the worker that already holds the KV blocks for its prefix, instead of round-robin and re-prefilling from scratch. The workload where this matters most is multi-turn chat: turn t’s prompt is turn t−1’s whole conversation plus one message, so the prefix-reuse rate approaches 100%. Route turn t to the worker that served t−1 and the prefill is one message long; route it elsewhere and the whole conversation re-prefills. This is the serving-side mechanism behind the prompt caching priced at the API level (prompt caching derived): a cache hit means the K/V blocks are already resident, and prefix sharing points the new sequence at them without copying.

At turn 20 of a chat that accumulates ~28,800 tokens of context, a cold worker on the 8B prefills all of it (~932 ms of TTFT) while a warm worker prefills only the ~1,200 new tokens (~39 ms), a ~24× TTFT gap that widens every turn. A router that lands 90% of turns warm versus 30% is worth more TTFT than any kernel optimization here, which is why the routing layer, the least GPU-specific component in the stack, is a performance component. The open example is the llm-d project (a Kubernetes-native scheduler scoring workers by prefix overlap); Dynamo ships an equivalent router.

The failure mode is a distributed-systems problem in GPU form: prefix affinity fights load balancing. Pin every conversation to its warm worker and hot conversations create a hotspot; balance perfectly and every hop costs a full re-prefill. Real routers score both (overlap × current load) and accept that the KV cache is now distributed state with the usual placement, eviction, and consistency problems, the same tension as a sticky-session tier.

Measuring: utilization, MFU, and the Nsight pair

Every claim so far was derived; on a live system you have to verify it, and the first tool most people reach for, nvidia-smi, measures almost nothing useful here.

Why “GPU util: 95%” is nearly meaningless

nvidia-smi’s utilization is kernel residency: the fraction of the sample window during which at least one kernel was executing, not how many SMs it used, not whether it computed or waited on memory. A kernel occupying 1 of 132 SMs but always resident reads “100%” while using ≤0.8% of the compute. This is not hypothetical: batch-1 decode keeps a kernel resident nearly always, so nvidia-smi reads ~100% while the chip does 0.34% of its possible arithmetic. “Util is high, we need more GPUs” is the wrong conclusion; the fix is batch size, which is free. The memory gauge misleads too: serving engines preallocate the KV space (vLLM grabs ~90% of HBM at startup), so “memory used” reads near-full whether the cache holds one sequence or seven hundred. Use the engine’s own counters for KV occupancy, and the two ratios below for efficiency.

The two ratios that mean something

• MFU, model FLOPs utilization: tokens/s × 2 × params / peak FLOP/s.
• MBU, model bandwidth utilization: bytes decode must stream per second (weights plus live KV) / peak bandwidth.

Which matters is phase-dependent, and that is the whole point. For an 8B at batch 1 measuring 146 tok/s, MBU is ~70% while MFU is ~0.24%: the same measurement, two denominators, opposite verdicts. This decode is healthy (70% of possible bandwidth, closing on the 209 tok/s ceiling) and its MFU is irrelevant. MBU also gives a distance-to-ceiling no counter does: 70% achieved means at most ~1.4× headroom on this workload, so a promised 5× kernel win here is impossible; the 5× must come from a lever that changes the workload (batch, quantization). For prefill it flips: a loaded 8B doing ~20,000 prompt tok/s is ~32% MFU, decent, with headroom worth profiling. Judge decode by MBU, prefill by MFU (40–60% is strong). Quoting the wrong ratio for the phase is the measurement version of quoting sparse TFLOPS.

What goes on the dashboard

The standing metrics that make either ratio computable, before you ever reach for a profiler:

• Tokens/s, prefill and decode separately. A combined number hides the phase split every diagnosis starts from.
• TTFT p50/p90 and ITL p50/p99. p99 ITL spiking with long-prompt arrivals is the head-of-line signature.
• MBU and MFU, computed from throughput and the fixed model constants, two multiplications, no profiler.
• KV-pool occupancy and preemptions/minute. Occupancy pinned at 100% with preemptions climbing means the batch is capacity-capped: the memory wall, not the scheduler.
• Batch occupancy per iteration. If it never nears its cap while the queue is nonempty, something upstream (admission, KV, token budget) is binding.

The Nsight pair

Two profilers at two levels, and the order matters more than any feature. Nsight Systems (nsys profile) is the system-wide timeline: every kernel, memcpy, and NCCL collective on one time axis. It answers “where did the wall-clock go?”: gaps (whitespace on the GPU row while the CPU is busy means launch overhead is pacing the GPU; the fix is CUDA Graphs, recording a step’s launch sequence once and replaying it as a unit), serialization (do NCCL collectives overlap compute, or alternate compute-silence-compute?), and foreign work on the critical path (host copies, tokenization on the serving thread). Nsight Compute (ncu) is the per-kernel view: it replays one kernel and reports why it is slow: the Speed of Light section (this kernel’s achieved percent of peak compute and peak memory side by side; a decode matrix-vector kernel at memory ~80%, compute ~5% is the roofline confirmed, while the same shape on a prefill GEMM is a bug), occupancy, and achieved bandwidth. The workflow is always nsys to find which kernel or gap owns the time, then ncu on that kernel to find why. Profiling kernel-by-kernel first is how people spend a week optimizing a kernel that was 4% of the step.

A healthy decode profile: timeline dense (gaps under a few percent, CUDA Graphs engaged); the step dominated by a few fused kernels, not thousands of slivers; those kernels’ Speed of Light showing memory high (60–80%+) and compute low (correct for decode, do not “fix” it); aggregate MBU 60–80%; NCCL under ~10% of the step and overlapped, at TP; batch pinned at the KV-memory or ITL limit, not an unraised default.

The levers, ranked

Everything above, indexed by symptom, because that is how the problem arrives:

Two rows deserve their arithmetic, both direct applications of the roofline.

Quantization stores weights in fewer bytes (INT8 or FP8, one byte per parameter) while computing in higher precision. Halve the bytes and decode intensity doubles, so on the memory roof the ceiling doubles: an 8B in INT8 is 8 GB of weights, ~2.39 ms/token, a ~419 tok/s ceiling (was 209). Weights-only is the safe first step (the KV cache and activations are separate, harder decisions), but it is the first lever here that can affect output quality. Measure on your evals before assuming the 2× is free.

Speculative decoding spends the FLOPs decode leaves idle. A small draft model proposes k tokens cheaply; the target verifies all k in one forward pass, and because that pass is memory-bound, verifying k+1 positions streams the same weights as generating one. With acceptance rate α, expected tokens per verify pass is 1 + α + α² + … + α^k: at k=4, α=0.7 that is ~2.77 tokens/pass, up to ~2.8× lower single-stream latency. The trade is explicit: total compute goes up (a whole draft model runs, rejected tokens are wasted) to reduce latency, the right trade on a machine that is 99.7% idle arithmetic, the wrong one on a compute-saturated prefill pool.

Order of operations, following production and cost’s discipline: measure first (MBU/MFU, the ITL distribution, KV usage and preemptions); then the free structural wins (paged KV and continuous batching, which a modern engine gives by default: confirm, don’t assume); then scheduling policy (chunked prefill, token budgets); and only then the levers that cost quality, hardware, or complexity (quantization, parallelism, disaggregation). Each step changes the numbers feeding the next, so the order is forced.

The mechanism → lever map

Cheat sheet

Numbers are representative (H100 SXM, BF16, spec-sheet peaks); re-derive on your SKU before quoting any.

Conclusion

• One ratio runs everything: ~295 FLOPs per byte on an H100. Decode sits at ~1 FLOP/byte, so it is memory-bound and runs at ~0.3% of peak; prefill sits at prompt-length, so it is compute-bound. Batching is the primary lever because it is the one that raises decode’s intensity toward the ridge.
• The KV cache decides batch size, and batch size decides throughput. Paged KV removes the fragmentation that capped batch; continuous batching keeps the batch full; chunked prefill keeps a long prompt from stalling the decode heartbeat.
• A 70B doesn’t fit on one card, so tensor parallelism (inside the NVLink domain, for latency and fit) and pipeline parallelism (across nodes, for throughput) are entry requirements, chosen by their communication arithmetic. At fleet scale, disaggregation and KV-aware routing let the two phases be sized and scaled independently.
• Measure with the right ratio: MBU for decode, MFU for prefill. nvidia-smi util is kernel residency and hides all of it; use engine counters plus nsys-then-ncu when you need to look inside a step.

Further reading

• Williams, Waterman, Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures” (CACM, 2009), the model this lesson is built on.
• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP, 2023), the vLLM paper.
• Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models” (OSDI, 2022), continuous / iteration-level batching.
• Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” (OSDI, 2024), chunked prefill.
• Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness” (NeurIPS, 2022).
• Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism” (2019), the tensor-parallel layout.
• Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving” (OSDI, 2024); Patel et al., “Splitwise: Efficient Generative LLM Inference Using Phase Splitting” (ISCA, 2024).