How to design an LLM serving system (a vLLM deep dive)

LLM inference is autoregressive, so serving is a batching-and-memory problem: keep the GPU busy across many requests while the KV cache stays inside num_gpu_blocks.

Two phases and the KV cache

• Prefill — whole prompt in one parallel forward pass; compute-bound (tensor cores saturate).
• Decode — one token per pass, streaming all weights from HBM; memory-bandwidth-bound (tensor cores idle).
• KV cache — cached key/value vectors of prior tokens; avoids quadratic recompute. It caps concurrency.
• kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × dtype_bytes. Llama-3-8B fp16 ≈ 131 KB/token; a 2,048-token chat ≈ 270 MB for one request.
• TTFT = queue + prefill. TPOT / ITL = decode step time. Chat wants low TTFT + steady TPOT; batch wants throughput.

Batching

• Decode is memory-bound, so serving one request at a time wastes the weight read. Batch: read weights once, emit a token for many.
• Static (request-level) batching — run N until all finish. Fails: head-of-line blocking, no mid-flight admission, padding waste. Utilization ~10–20%.
• Continuous batching (Orca, iteration-level) — every step, evict finished, admit waiting. What vLLM does.
• Selective batching — flatten sequences into one “super-sequence” for position-independent ops (matmuls, MLP, layernorm); compute attention per request via a mask. No padding.

PagedAttention (vLLM KV manager)

• Pre-vLLM: one contiguous max-length KV region per request → internal + external fragmentation, no sharing.
• KV cache = pool of fixed physical blocks of block_size tokens (default 16); each request has a block table mapping logical positions → non-contiguous physical blocks; blocks allocated on demand.
• Fragmentation drops to <1 block/sequence → pack more sequences → 2–4× throughput at equal latency vs Orca-gen systems.
• Prefix caching — same prompt prefix points block tables at the same physical blocks. Parallel sampling / beam — branches share prompt blocks copy-on-write.
• slot_mapping tells the kernel which physical slot each token’s K/V goes to.

The engine (vLLM V1)

• LLMEngine / AsyncLLM — tokenize, build requests, detokenize/stream.
• EngineCore — owns scheduler, KV manager, executor.
• Scheduler — waiting + running queues; picks requests and per-request token budget each step.
• KVCacheManager — free_block_queue, req_to_blocks, prefix-cache hash table.
• Executor / Worker / ModelRunner — forward pass, weights, KV tensors, sampler. UniProcExecutor = 1 GPU; MultiProcExecutor = TP/PP.
• Init profiling: measure weights + peak activations, then num_gpu_blocks = kv_budget // block_bytes. That one number is the hard ceiling on concurrency × length.

step():  scheduler.schedule() → executor.execute_model() → sampler.sample() → postprocess()
schedule(): decodes first (1 tok each, cheap) → then chunked prefills up to token budget
allocate_slots(): pop from free_block_queue; new block only when length crosses a 16-tok boundary; else preempt

Latency and scaling knobs

• Chunked prefill — cap prompt tokens per step (long_prefill_token_threshold) so a huge prefill doesn’t stall everyone’s decode. Slightly higher TTFT for the long prompt, smooth TPOT for all.
• P-D disaggregation — prefill and decode on separate worker pools; prefill writes KV to a connector, decode loads via start_load_kv.
• Prefix caching — chain-hash each full 16-token block h = hash(prev_h, token_ids, lora_id); only full blocks cacheable; ref-counted, returned at count 0; on by default. A one-char prompt edit invalidates every cached block.
• Speculative decoding — cheap draft proposes k tokens, target verifies all in one pass, accept longest correct prefix (distribution-preserving). Variants: n-gram, EAGLE, Medusa. Up to 2–3× TPOT on predictable text.
• Scaling: tensor parallelism (shard layer matrices, all-reduce/layer), pipeline parallelism (layer ranges + micro-batch), data parallelism (replicate engines; route by len(waiting)*4 + len(running)).

Preemption and the roofline

• Out of blocks → preempt a victim. Recompute (drop KV, re-prefill on resume; cheap memory, costs compute; default short seqs) vs Swap (KV to CPU RAM and back; costs PCIe, saves recompute; better long seqs).
• Roofline: below B_sat step time is flat (weight-read dominated) so more requests are near-free throughput; above B_sat it’s compute-bound and grows with B, raising TPOT.
• Tune gpu_memory_utilization, max_num_batched_tokens, max_num_seqs against the curve. Optimize goodput (requests within SLO), not raw tokens/sec.

Gotchas / detectors

• KV exhaustion / preemption thrash → track preemption count + queued:running; alert before KV utilization = 100%.
• Long-prefill head-of-line blocking → watch p99 TPOT (not mean); enable chunked prefill.
• Length creep → alert when avg sequence length rises while running-batch size falls.
• Prefix-cache stampede → watch hit-rate after any prompt/template change.
• Bench: vllm bench latency (ITL), throughput (QPS=∞), serve (Poisson arrivals).