How to design an assistant chatbot

An assistant has no ground truth, so it lives on two decisions: define “good” as separate trading axes, and size the fleet from KV-cache bytes, not compute.

Problem framing

• Turn: one user message + reply. Token: ~4 chars of English. TTFT: wait before output starts.
• Constraints: first token < 500 ms, > 30 tok/s sustained, tens of millions DAU, cost/turn that survives a free tier.
• The model has no memory between calls, so the whole conversation is resent every turn. State, not compute, dominates cost.
• Evaluation set must be stratified by real traffic mix; a 70%-coding set for 22%-coding traffic measures the wrong product.

Quality: six axes, never averaged

• Derived backwards from failure traces; add an axis when a new trace fits none. Sycophancy = honesty fail a helpfulness metric rewards; over-refusal = safety metric moving right while product worsens.
• Report violation rate and false-refusal rate separately; averaging destroys the signal.

Training stack

• Pretraining = capability. SFT (30k–100k demos) = format + tone; style dominates because it is high-frequency, facts are one-off. Trained behavior cannot be suppressed by system-prompt text (advisory ≠ enforcement).
• DPO vs RLHF+PPO: DPO holds 2 models (policy, ref), consumes pairs, hours/stable, cannot improve past the data. PPO holds 4 (policy, ref, reward, value), can improve past data via RM generalization, days/unstable, needs KL tuning.
• Switch rule: ship DPO, move to online RL only when the RM’s held-out agreement with experts beats inter-annotator agreement (the ceiling on everything downstream).
• Flywheel: weak signals (thumbs, regenerate, edits) select, they do not reward — thumbs-down agrees with experts at 0.55 (~coin flip). Route the ~0.5% of traffic where signals disagree to paid annotators.

Conversation state and cache

• Attention is causal: a cached token is valid only while every token before it is unchanged.
• Rule 1: nothing volatile before anything stable (put the date in the current turn, not the system prompt — fails silently, just a bigger bill).
• Rule 2: user memory is a per-user prefix, placed after shared system+tools, else every user caches a private copy of 1,500 tool tokens.
• Compaction at 70% of working budget: keep first message + task + last 6 turns + pinned constraints verbatim, summarize the rest, re-anchor task at the end. Invalidates cache from the split point (one re-prefill), then back to cheap append.
• Compact at a threshold, never incrementally (sliding window changes the prefix every turn = re-prefill forever). Cost of not compacting is quadratic: n·(sys+tools) + a·n²/2; 40 turns ≈ 16× a 10-turn one.

Safety: enforcement is in the harness

• Advisory = model + system prompt (a tendency). Control = enforced regardless of the model: two classifiers + tool gate.
• Prompt rules can’t enforce: 900 system tokens vs up to 6,000 user tokens in one undifferentiated sequence, ~7:1 attacker-controlled. Must be false in Python.
• Input classifier: ~30 ms, narrow hard-block only (harm is intended, ambiguous). Output classifier carries the spend (harm is manifest): 200-token rolling windows + 50-token lookback, 50-token buffer behind the stream. Cost ≈ 50/40 ≈ 1.2 s, hidden by streaming; a full-response pass still closes the turn.
• Jailbreak resistance is trained (adversarial pairs), gated on a 1,200-case red-team set refreshed on a schedule (goes stale after ~2 rounds).
• Capability set frozen at turn start from trusted state, before any external content. Blocks the lethal trifecta (private data + untrusted content + external write). Check the materialised context, raise (not assert — python -O strips it), and use a frozenset (substring check on a string silently authorizes).

Serving: fleet is set by KV bytes

KV bytes/token = 2 × n_layers × n_kv_heads × head_dim × bytes

• 70B, 80 layers, head_dim 128, fp16: MHA (64 KV heads) = 2.62 MB/token; GQA 8:1 (8 KV heads) = 320 KB/token. Decided at pretraining, frozen in the weights.
• Node = 4× H100 (80 GB) tensor-parallel: 140 GB weights + 20 GB activations + 160 GB KV = the capacity.
• 8k-token avg context → 2.62 GB/session → 61 resident. Share prefix (paged attention + copy-on-write): 0.79 GB once + 1.84 GB/session → 86 resident (+41%). Duty cycle (20 s TTL, warm-hit 1−e^(−20/25)=0.55) → 86 × 1.39 = 120 concurrent.
• Little’s law L = λW: 10M DAU × 3 = 30M sessions, λ=347/s, W=480 s → 166,667 avg × 2.2 peak → 366,667 / 120 = 3,056 nodes = 12,222 GPUs. Round once, at the end.
• At $2.50/GPU-hr: $267.7M/yr, cost/turn ≈ $0.002 (16× under $0.033 list), $2.20/DAU/month.

Lever table

• GQA is a record, not a lever — frozen at pretraining ($268M vs $2.2B). fp8 = best ROI, gate on long-context recall (degrades there first).
• Fleet ∝ conversation tokens (prefix paid once): +1,000 tokens ≈ 18% of fleet (~$48M/yr).
• Route models at session start, escalate small→large once, never de-escalate — KV caches are model-scoped, so a mid-conversation switch throws the whole cache and re-prefills.
• Continuous batching 4.3× (static utilization 1/H_64 ≈ 21% → ~90%). Speculative decoding: draft k=5, quote 2.94/1.5 ≈ 1.96× not 2.94×; enable below 50% batch occupancy, off at peak. Streaming is the product: 600/40 = 15 s, no hardware makes one sequence faster.

Metrics traps

• Judge attenuation: an LLM judge at agreement a compresses effects by (2a−1); at 0.85 a true 4-pt gap reads 2.8 — 0.85 is a floor.
• Single number lies: B wins 53.6% overall yet loses on 31% of traffic; and length confound — 2.8 of 3.6 points were verbosity (reward hacking). Report length-controlled.
• D7 return rate: the only metric no within-turn behavior can game (but slow, noisy). A/B: randomize per user (never per turn), keep a permanent 1% retention holdout, keep guardrails (p99, false-refusal, hallucinated-citation) as independent blockers.

Failure modes → fix lives at a different layer

Prereq for every diagnosis: full prompt capture per turn + replay against a fixed checkpoint. Read the prompt sent, not the transcript.