LLM Inference Performance

Every serving decision follows from one ratio: how much arithmetic an H100 can do per byte it pulls from HBM (~295 FLOPs/byte); decode falls far below it and is memory-bound, prefill sits above it and is compute-bound.

The ratio and the roofline

• FLOP = one floating-point op. Ridge point = peak compute ÷ HBM bandwidth = 989e12 / 3.35e12 ≈ 295 FLOP/byte (H100 SXM, dense BF16).
• Below the ridge: memory-bound (throughput set by bytes moved). Above it: compute-bound (arithmetic units are the wall).
• Arithmetic intensity = FLOPs per byte moved from HBM. attainable FLOP/s = min(peak compute, intensity × bandwidth).
• Gotcha: NVIDIA’s headline 1,979 TFLOPS assumes 2:4 sparsity. Served weights are dense → use 989. Sparse figure halves the ridge, doubles apparent efficiency.
• Sweeping all 80 GB HBM once ≈ 24 ms → any full-card step capped at ~40 steps/s.

Prefill vs decode

• Forward pass ≈ 2 FLOPs per parameter per token; each BF16 param = 2 bytes.
• Decode (batch 1) intensity = 2 FLOP / 2 B = 1 FLOP/byte → min(989e12, 1×3.35e12) = 3.35e12 = 0.34% of peak. Not misconfig; the workload can’t feed the units.
• Prefill intensity ≈ n_tokens FLOP/byte; 2,048-token prompt → compute-bound. Prompt shorter than ~295 tokens is still memory-bound.
• Batching raises decode intensity: intensity ≈ B, break-even at B ≈ 295. But KV cache does not amortize (streamed in full per sequence per step), so effective intensity drops at long context.

Latency metrics and head-of-line blocking

• TTFT = queue + full prefill (delay before first token). ITL = gap between streamed tokens (spikes = stutter). TPOT = mean of ITLs.
• Head-of-line blocking: a naive scheduler runs a long prompt whole, stalling every in-flight decode (8K prompt ≈ 260 ms freeze vs 6 ms baseline). Watch p99 ITL, not the mean.

KV cache sizing

• bytes/token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes. Linear in context.
• GQA (query heads share K/V) and MQA (one shared K/V) shrink n_kv_heads; both are training-time choices. Trap: using query-head count (64) for a GQA model inflates the answer 8×.
• 70B GQA fp16 = 320 KB/token → 8K seq = 2.62 GB; 128K ctx = 41.9 GB (>½ a card in one conversation).
• Max concurrency = free HBM ÷ per-seq KV. 8B: 60 GB free / 1.05 GB ≈ 57 worst-case 8K seqs — memory runs out before batch reaches the ridge (295).
• Decode cost per sequence degrades with context: crossover where one conversation’s KV costs as much bandwidth as the whole model ≈ 122,000 tokens (8B).

PagedAttention and batching

• PagedAttention = OS virtual memory for KV: fixed blocks (~16 tokens), per-sequence block table, allocate on demand. Fragmentation collapses to ≤1 partial block/seq (~2 MB vs ~1 GB reservation). Pre-paged systems wasted 60–80% of KV memory.
• Prefix sharing (block tables point at shared blocks) + copy-on-write kill duplication; preemption evicts/recomputes blocks when memory runs out (nonzero rate = KV budget is the constraint).
• Payoff is batch: same 8B card holds ~57 seqs naive → ~760 paged → ~3.6× throughput. Trade: bigger batch raises ITL (6.1 → 22.6 ms), so schedulers cap batch to hold an ITL target.
• Static batching: pad to equal length, drain fully — wastes on length variance. Continuous batching (Orca, iteration-level): finished seqs exit and new ones join at token boundaries. Chunked prefill (Sarathi): split prompt into ~512-token chunks co-scheduled with decode, trading prefill efficiency and long-prompt TTFT (+40%) for smooth ITL (260 → 23 ms).

Parallelism

• 70B BF16 = 140 GB > 80 GB card → multi-GPU is an entry requirement, not an optimization. Replicas scale request throughput but don’t help fit or per-token latency.
• Tensor parallel (TP): splits every matrix across p GPUs; 2 all-reduces (NCCL) per layer per token. Bytes trivial, but 70B TP4 = 160 collectives/token × ~5 µs ≈ 0.8 ms (NVLink) vs ~4 ms (PCIe). Only scheme that cuts per-token latency (memory wall ÷ p). Keep TP inside the NVLink domain (one node).
• Pipeline parallel (PP): contiguous layer stages, one activation handoff per boundary → crosses nodes. Bubble fraction = (p−1)/(m+p−1); p=4, m=16 → 15.8%, m=64 → 4.5%. Throughput-only; doesn’t cut single-stream latency. Keep microbatches ≫ stages.
• Standard layout: TP as far as NVLink reaches (~8 GPUs), PP across nodes only when a node is still too small.

Disaggregation and KV-aware routing

• Disaggregated serving (DistServe/Splitwise; NVIDIA Dynamo): separate prefill and decode pools, each sized/scaled on its own bottleneck (prefill = prompt-tok/s; decode = concurrent-seqs-at-ITL). Interference gone by construction. Optimize goodput (req/s meeting both TTFT and ITL SLOs), not raw tokens/s.
• KV hop cost: 2.62 GB over NVLink ~3 ms, over 400 Gb RDMA ~52 ms; hidden by layer-by-layer overlap. Below node scale, share; at fleet scale, split.
• KV-cache-aware routing: send a request to the worker already holding its prefix blocks. Multi-turn chat has ~100% prefix reuse; warm vs cold at turn 20 ≈ 39 ms vs 932 ms TTFT (~24×). Failure mode: prefix affinity fights load balancing → score overlap × current load.

Measuring

• nvidia-smi util = kernel residency (fraction of window with ≥1 kernel running), not SM count or compute. 1 of 132 SMs busy reads 100%; batch-1 decode reads ~100% at 0.34% arithmetic. Memory gauge also misleads (engines preallocate ~90% HBM).
• MFU = tok/s × 2 × params ÷ peak FLOP/s — judge prefill (40–60% strong). MBU = streamed bytes/s ÷ peak bandwidth — judge decode (60–80% healthy). MBU also gives distance-to-ceiling: 70% → ≤1.4× headroom.
• Nsight Systems (nsys) = system timeline (gaps → CUDA Graphs; NCCL overlap; foreign work on critical path). Nsight Compute (ncu) = per-kernel, Speed of Light (memory ~80% / compute ~5% confirms decode roofline). Order: nsys to find the costly kernel/gap, then ncu on it.

Levers by symptom

• Quantization: INT8/FP8 (1 B/param) doubles decode intensity and ceiling (8B INT8: 209 → ~419 tok/s). Weights-only is safe first step but can affect quality — measure on evals.
• Speculative decoding: draft proposes k tokens, target verifies all in one memory-bound pass. Expected tokens/pass = 1 + α + … + α^k (k=4, α=0.7 → 2.77, ~2.8× lower latency). Spends idle compute — wrong on a saturated prefill pool.
• Order of operations: measure → free structural wins (paged KV, continuous batching) → scheduling (chunked prefill) → cost-bearing levers (quantization, parallelism, disaggregation).