How to design smart compose

Inline sentence completion for email (grey text, Tab accepts): a latency-and-precision problem, not a text-quality one. Two numbers settle the design: the 100 ms keystroke budget picks the model, and the cost asymmetry picks the show/hide threshold.

The two governing numbers

• Latency budget: total time fixed before design; every stage draws from it. Here p99 100 ms keystroke-to-pixels, set by a fast typist’s ~150 ms inter-keystroke gap (80 wpm). A suggestion arriving after the next keystroke is worse than silence (wrong prefix, wasted eye fixation).
• Cost asymmetry: a wrong suggestion costs strictly more than nothing. The trigger threshold is derived from that gap, not tuned.
• Output nothing is valid and most common: suggestion shown on only 12-14% of consulted moments.
• Volume: 50M DAU, ~6 composes/day of ~340 chars; debounce → ~48 model evaluations/compose.

Model size, from the budget alone

Subtract fixed costs, then each model is one division (batch 64, 700-token context, 3.3 TB/s H100):

• Server decode budget ≈ 25 ms (100 − 40 debounce − 34 network − 2 render); on-device ≈ 58 ms (no network).
• Decode is memory-bandwidth-bound: step_time = (weight_bytes + batch·context·kv_per_token) / bandwidth. Ranking set by weight bytes, so hardware changes every time but not the order. Quality never enters; big models excluded first.
• Survivor: decoder-only transformer, ~297M params (24 layers, d_model 1024, 16 query / 4 KV heads GQA, vocab 32k, tied embeddings), KV = 24 KB/token. Trained from scratch on mail corpus.

Decoder-only, not encoder-decoder

• Causal attention: token i’s K,V depend only on tokens 0..i. Appending a token leaves all prior K,V bit-identical → cacheable.
• Bidirectional encoder: every representation depends on every token, so one keystroke re-encodes all 700.
• The win is the 4-vs-700 ratio: 700/4 = 175x less prefill fleet-wide. Holds while context ≈ 700 tokens; at 4,000 the KV term dominates → separate server tier at relaxed 400 ms budget.

The two objectives (never conflate)

• Generation: next-token cross-entropy L = −Σ log P(x_t | x_<t, context). Perplexity = exp(avg cross-entropy) = a gate, not a goal; an 8%-better perplexity that never fires the trigger ships zero chars.
• Decision: calibrated binary choice with asymmetric costs. Needs a usable calibrated confidence from the 300M’s outputs (buckets scored 0.7 → ~70% right). Enables selective prediction (answer only when confident). Cheap check: plot observed accuracy vs predicted confidence; diagonal = works, flat = kills it.
• Small model is defensible because it stays silent: 300M/7B top-1 agreement is 71% over all positions but 96% on the confident 12% slice where the trigger fires.

The derived threshold

Utile = one average accept (18 chars). Show when expected value positive: q·A_ok·V_a > (1−q)·(C_r + A_wrong·C_w).

• Priced: A_ok=0.55, A_wrong=0.03, V_a=1.00, C_r=0.05, C_w=30.
• C_fp = 0.05 + 0.03·30 = 0.95; C_fn = 0.55·1.00 = 0.55.
• q* = C_fp/(C_fp+C_fn) = 0.95/1.50 = 0.633.

• The 0.083→0.633 gap (7.6x) is entirely the A_wrong·C_w term. A_wrong is the most sensitive number: 0.005 → 0.27, 0.10 → 0.85. First launch is conservative, to measure it.
• Threshold needs a calibrator (tiny logistic/GBDT on output features), not raw sequence probability (uncalibrated + length-confounded → silently caps length at 5 tokens).
• Length is the same decision: sweep L, accumulate q(L). Unconditional optimum 2 tokens; on the fired slice it moves to 5-6 tokens, peak utility +0.082 → +0.402 (~5x). That gap is the value of the trigger.

Privacy is the data architecture

Corpus is others’ private mail; make failures impossible, each mechanism carries a measured cost.

• DP guarantee = (ε=8, δ=1e-9) under RDP (Rényi) accounting, at subsample rate q=20k/50M=4e-4, 3,000 rounds, σ=1.0. Amplification by subsampling carries most of it; relative noise falls as 1/m, so DP only affordable at scale.
• δ must sit well below 1/N (1/50M=2e-8; 1e-9 clears by 20x).
• DP blocks memorization; measure with canary insertion in bits of exposure. A secret appearing 8x in 300M msgs: 14 bits (extractable) undefended → below floor with DP.
• Under ~200k users this is a different design: no federated training, no whitelist, filtered public corpus, broad suppression.

Trigger pipeline (four gates)

keystroke → seq# ↑ → debounce (40ms quiet + word boundary?)
  → hard suppression → 300M int8 (KV warm) → calibrator q
  → q ≥ 0.633 AND U(L) > 0? → seq# still current? → render grey

• Hard suppression runs before the model, in code: numbers/currency/dates, gendered pronouns, URLs/emails/phones, entities absent from context, protected-attribute proximity (12 tokens), off-whitelist strings.
• Pronoun rule falls out of the same formula: V_a=0.17, C_w=400 → q*=0.992, unattainable → suppress the class.
• Debounce: word-boundary + 40 ms quiet cuts 100 keystrokes → 14 requests (word-boundary-only = 22). The 40 ms gate sits in the bimodal trough (intra-burst 20-30 ms vs inter-burst pauses); derive it from your own histogram, never the 150 ms mean.
• Cancellation mandatory: stale seq# discarded on client, in-flight server request cancelled — else a suggestion fades in after the user typed past it.

Why on-device

• Not primarily privacy — it is a memory wall. Server KV residency: ~1.07M peak sessions × 17.2 MB = 18.5 TB → 231 accelerators holding bytes vs 20 decoding (~11x more hardware for cache than compute).
• On device: one session, 17.2 MB is nothing, cache is free, lives next to the only allowed viewer.
• p50 differs by ~14 ms (decides nothing); p99 by 132 ms (device 78 ms / thermal, bounded & yours; server ~210 ms / network tail, unfixable). p99 is the budget.
• Phone bandwidth ~68 GB/s (LPDDR5X) vs 3.3 TB/s (~48x less) caps model: 300M int8 = 4.41 ms/tok (fits), 1B = 14.7 ms/tok (does not). Segment latency guardrail by device class; old handsets fall back to n-gram model.

Metrics

• Offline headline: exact-prefix precision (acceptable set has size 1 — a single Tab; BLEU/ROUGE actively wrong). Perplexity is a gate only.
• Counterfactual replay: only offline number tracking online, because it exercises the decision layer. Systematically optimistic; ratio observed/simulated = A_ok (316/566 = 0.56). Calibrate once vs A/B, use as gate not forecast.
• Online headline: chars saved/user/day = 316 (~15.5% of typing). Acceptance rate is derived (0.78·0.55 + 0.22·0.03 = 0.436), not measured; failure to reconcile = valuable signal.
• Acceptance rate misleads: you control the denominator (0.633→0.90 raises acceptance 44%→52% while chars saved falls 316→68); no opted-in impression; rejection has negative price CTR reads as zero.
• Bad accepts (~1 every 4 days) eat 45% of gross value (8.0/17.6 utiles) — the whole case for precision-first.
• Guardrails block launch on regression: feature-disable <0.3%/mo, retraction rate <4% (free read on A_wrong, instrument first), suggestions/1k keystrokes, no typing-speed regression, p99 <100 ms.
• A/B randomize on the user (trust/disable accumulate per person); read at week 3 not week 1 (novelty). ~12,300/arm (~7,400 with CUPED); binding constraint is time, not users.

Gotchas / failure modes

• Factually wrong completion (invoice total at p=0.71) → numeric suppression; retraction spikes on currency tokens.
• Biased pronoun from job title → the q*=0.992 suppression, same formula, different C_w.
• Memorized personal data → canary suite detects; DP + whitelist + numeric suppression guard.
• Stale suggestion race → sequence-number check (five-line fix; its absence is the commonest way it feels broken).
• Cross-session cache collision → key on (session_id, prefix_hash), not prefix alone.
• Non-Latin scripts degrade silently (tokenizer fragments → fewer words/suggestion) → segment every metric by language, per-language length policy.
• Feature eats its own training data: mail gets templated, suggestions blander, acceptance rises while value falls → hold a permanent control cohort, monitor KL divergence of sent-mail n-grams.

Cost, and the one line

• On-device: $0 marginal (300 MB binary, ~8 s NPU/day). Server 300M with cache reuse: ~$5.6M/yr; without: 3.7x ($21M/yr). Hosted frontier API: ~$19.2B/yr (input prefix is 96% of the bill; disqualified on latency first, cost second).
• One line: a latency-and-precision problem wearing a text-quality costume — let the 100 ms budget pick the model and the cost asymmetry pick the threshold.