How to design smart compose

Smart Compose finishes your sentence while you type an email. You write half a sentence, grey text appears proposing the rest, and pressing Tab accepts it. The task: build inline sentence completion for an email client. As the user types, suggest the rest of the sentence in grey; Tab accepts, any other key dismisses.

This is not a text-quality problem. It is a latency and precision problem, and two numbers settle almost the entire design:

1. How long you have before the next keystroke. The budget is one gap between keystrokes, about 100 ms at the 99th percentile. That disqualifies most models before quality is ever discussed.
2. How much more a wrong suggestion costs than showing nothing. A wrong suggestion costs strictly more than silence, so the threshold that decides when to show one is derived from that asymmetry, not tuned.

In this lesson, we’ll do four things, in order: size a model from a latency budget instead of a benchmark; derive the confidence threshold that decides whether to show anything at all; explain why the feature runs on the phone instead of in a data centre; and build the privacy machinery that makes training on private mail possible. By the end you’ll be able to eliminate a model with one division against the budget, compute the show-or-stay-silent threshold from the cost asymmetry, and defend the on-device choice from memory arithmetic.

Terms used throughout

A latency budget is the total time the system is allowed to take. You fix it before designing anything, then spend it: every stage of the pipeline draws from the same pot, and when the pot is empty the design is over. Sizing a model from a latency budget means you do not pick a model and measure its speed; you decide what you can afford and let the budget eliminate models before quality enters the conversation.

Prompt caching matters here for one reason: the cached state for a token depends only on the tokens before it. That single property is what the whole architecture is chosen to preserve, and the decoder-only argument is nothing but that property spelled out.

Problem framing

The most important fact about the output is that nothing is also a valid output, and it is the output most of the time. The suggestion is shown on only 12-14% of the moments where the system is consulted (derived later); on the other 86-88% the system deliberately shows nothing. A design that treats “produce a continuation” as the task has missed the product.

Of the questions that usually shape a design like this, two dominate here and the rest barely register:

• The latency budget overrides everything. It is so tight that it eliminates most models before quality is mentioned.
• The cost asymmetry sets the operating point. The gap between a wrong suggestion and no suggestion is what the trigger threshold is computed from.
• Cost-per-request and downstream schema do not matter: the output is user-facing prose with no parser to reject it, and every surviving option meets the cost ceiling easily.

Where the 100 ms budget comes from

The number the rest of the design rests on is the interval between one keystroke and the next, the inter-keystroke interval, for a fast typist. Typing speed is quoted in wpm (words per minute, where a “word” is a standard five characters). Converting to a gap per keystroke:

average typist   40 wpm  ->  3.3 chars/s  ->  300 ms between keys
fast typist      80 wpm  ->  6.7 chars/s  ->  150 ms between keys

A suggestion that arrives after the next keystroke is not merely late but wrong, because the prefix it was conditioned on no longer exists. It renders, the user’s eye moves to it, and it disappears on the next character. That is worse than showing nothing, because it cost the user a fixation, one deliberate movement of the eye onto the grey text, and returned no value.

So the budget is the fast typist’s interval, with margin for the work that is not the model: 100 ms at p99, keystroke to pixels. The p99 is set by the fastest users, not the average one, and fast typists are the heavy composers.

Every later decision (the 300M model, the decoder-only architecture, the on-device deployment, the debounce) is a consequence of this 100 ms. The one subtlety worth checking is how often the deadline is missed. At 48 evaluations per compose and 1-in-100 misses, most misses land on an evaluation that was going to show nothing anyway, so they are invisible. Weighting by the 14% chance any evaluation renders, the user sees roughly one late render every fifteen messages: frequent enough that a heavy composer meets it weekly, rare enough that no dashboard surfaces it.

The two objectives

The system learns two things: what to say, and whether to say it. Conflating them is the most common error in this problem.

Generation is the standard next-token objective. Penalise the model in proportion to how surprised it was by the token that actually came next. That penalty is cross-entropy (the negative log-probability the model assigned to the true token, summed over the sequence) and minimising it means maximising the probability the model gives to text the user really wrote:

L  =  - sum_t  log P(x_t | x_<t, context)

where x_t is the token at position t, x_<t is everything before it, and context is (recipient, subject, thread, prefix). A confident correct prediction contributes almost nothing; a confident wrong one contributes a lot.

The decision is separate, and it is where the product lives. Given a candidate continuation and the model’s probabilities, decide whether to show anything and how long the suggestion should be. This is a calibrated binary decision with an asymmetric cost matrix:

• Calibrated means the number the system calls a probability behaves like one: among cases scored 0.7, roughly 70% turn out right.
• Asymmetric cost means the two ways of being wrong (showing a bad suggestion, withholding a good one) do not cost the same.

No amount of language-modelling progress solves the decision problem. Perplexity is a gate, not a goal: it is the exponential of average cross-entropy, readable as the number of equally-likely options the model was choosing between at each token. A model with 8% lower perplexity that fires the trigger no more often ships zero extra characters saved. The product metric is characters saved per user per day at a fixed annoyance budget, and that is the product of the language model and the decision layer.

The decision layer depends on one thing being possible: extracting a usable calibrated confidence from a 300M model’s own outputs. If the model is no more often right when it claims 0.9 than when it claims 0.5, then selective prediction (answering only where you are confident, staying silent otherwise) is impossible, and the small model can no longer hide behind a confident slice. The cheap check to run first: bucket held-out positions by predicted confidence and plot observed accuracy against it. A line near the diagonal means the design works; a flat line kills it.

Data and labels

Every sent message is a completed sequence, so supervision is free and enormous: the user already wrote the right answer. Take a random cut point in each message: everything before it is the prefix, the next N tokens are the target.

sent:   "Thanks for sending that over. I'll review it and get back to you Friday."
cut ->  prefix:  "Thanks for sending that over. I'll review"
        target:  " it and get back to you"

Raw sent mail is not training data. Each pipeline stage removes something specific, and skipping it leaves a specific failure. (An n-gram is a run of n consecutive tokens; counting n-grams is how you find text that repeats across a corpus.)

After filtering the corpus is roughly 300M messages with 4 cut points each, so about 1.2B training examples. The average surviving message is ~340 characters, so at 4 chars/token about 85 tokens. That gives two defensible token counts: ~8B tokens if you count only the predicted spans (1.2B examples × ~6.7-token targets), or ~25.5B if you count every token read (300M × 85). Which belongs in a cost estimate depends on whether you pack whole messages and compute loss everywhere (25.5B) or sample short targets (8B). The training cost is $33 at 8B and $106 at 25.5B, rounding error against the pipeline and calibration work either way, so the ambiguity is not worth resolving.

The load-bearing assumption is that sent mail is the right distribution to imitate, and every filter enforces that: machine-generated and pasted text are removed because nobody typed them. If the corpus were contaminated with text no human composed, the model would be fluent at exactly the sentences the feature will never assist, and this would not show up in perplexity, because perplexity is measured on the same contaminated held-out set. The diagnostic that catches it is per-position accuracy on real live prefixes; the two coming apart is the signal that the corpus has drifted from traffic.

Privacy is the data architecture, not a compliance appendix

The corpus is other people’s private mail, so the architecture has to make certain failures impossible, not merely unlikely. Four mechanisms, in increasing strength, and each carries a measured quality cost. A privacy mechanism quoted without its utility cost is a slogan.

1. Never log the prefix. The request payload is the user’s unsent email, so a debug log of requests is a mail archive nobody signed off on. Log token counts, latencies, the trigger decision, and the outcome, never the text. This is free, and it is the one teams actually get wrong.

2. Federated learning trains a shared model without collecting the training data. Each device trains on the mail it already holds and sends back only an update (dW): the small weight change that would make the model better at predicting that user’s mail, derived from the gradient (the direction of steepest improvement). Updates are combined under secure aggregation, a cryptographic protocol that masks each client’s contribution so the server can compute the sum but cannot read any single term. Net effect: data never leaves the device, and the server sees only a sum over at least 20,000 clients per round.

flowchart TD
    subgraph DEV["On device — data never leaves"]
        D1["Local sent mail"] --> D2["Compute update dW<br/>on the local corpus"]
        D2 --> D3["Clip dW to length S<br/>(bound one user's influence)"]
    end
    D3 --> AGG["Secure aggregation<br/>server sees only the SUM<br/>over m >= 20,000 clients"]
    AGG --> NZ["Add Gaussian noise<br/>proportional to S"]
    NZ --> UP["Average and apply<br/>to the global model"]
    UP -.->|next round| D2

3. Differential privacy (DP) is a formal guarantee that the trained model would come out very nearly the same had any one user’s data been absent, so nothing user-specific can be recovered from it. It is delivered by two steps, and neither works alone:

4. Clip each client’s update to L2 norm S (its geometric length as a vector). No client can push the model further than S, however unusual their mail. Noise without clipping is defeated by one outlier whose enormous update swamps it.

5. Add Gaussian noise to the aggregate, scaled to S. Clipping alone still leaks, because a bounded contribution is still a contribution you can look for.

DP is affordable only at scale. The noise added to the sum is the same size regardless of how many clients contributed, while the signal grows with the number of clients, so the relative noise falls as 1/m in clients per round. Millions of users is exactly the situation where you both can afford DP and most need it.

A DP guarantee is quoted as (epsilon, delta): epsilon caps how much more likely any observable outcome becomes because one user was included (smaller is stronger; 8 is a moderate real-world setting), and delta is the small probability the bound fails (here one in a billion). Two things produce that number and it is meaningless without them:

• The accountant. Composing 3,000 rounds naively multiplies the per-round loss into the hundreds. A Rényi differential privacy (moments) accountant tracks the loss as a Rényi divergence and converts once at the end, making growth in rounds behave like sqrt(T) instead of T.
• The subsampling rate q: the fraction of the population touched per round, here 20,000 / 50M = 4e-4. (This q is DP-literature notation and is unrelated to the calibrated confidence q used from the trigger policy onward; the collision is universal.) Privacy amplification by subsampling is most of where the guarantee comes from: a user not sampled in a round incurs no loss that round. Raise the rate tenfold and the guarantee degrades sharply even with every other symbol unchanged.

So the honest form is five numbers: (epsilon = 8, delta = 1e-9) under RDP accounting, at q = 4e-4, over 3,000 rounds, at noise scale sigma = 1.0. One sanity check: delta should sit well below 1/N, because a delta near 1/N permits leaking one whole user. Here 1/50M = 2e-8, so 1e-9 clears it by 20x. Measured cost: +3.1% perplexity, which propagates to -1.4 points of acceptance rate.

What DP buys is protection against memorization, a model reproducing a rare training string verbatim instead of generalising. Given my account code is , an undefended model trained on raw mail will complete it with a real code belonging to a real user. You measure this with canary insertion: plant a synthetic secret of the right shape a controlled number of times before training, then measure how strongly the model prefers the true canary over random alternatives, reported in bits of exposure (14 bits means it narrowed ~16,000 candidates to one; “below floor” means indistinguishable from chance).

The middle row is the point: a secret appearing just 8 times in 300M messages is recoverable from an undefended model, and DP pushes it below the detection floor. A canary suite is the only thing that turns “we take privacy seriously” into a number, and it runs on every training run like any other test.

4. k-anonymity on the output side. Never emit a string unless at least k different people independently wrote it, so no output traces back to one person. With k = 50, maintain a whitelist of n-grams that appear in the sent mail of at least 50 distinct users, and refuse any suggestion not on it. This is enforcement, a branch in code, not advisory, a rule written into the prompt that only shifts the model’s logits (advisory vs enforcement). It makes the memorization failure unrepresentable, not merely unlikely. Measured cost: coverage falls ~9%, because idiosyncratic-but-correct completions get blocked too. That is the right trade.

Every mechanism here scales with population. At 200,000 users the DP noise would swamp the signal at a comparable epsilon, and the whitelist would be so sparse that coverage collapses. A small-population design is a different design: no federated training, no whitelist, an aggressively filtered public corpus, and much broader suppression.

Model choice, from the latency budget

The model size is settled by arithmetic on the latency budget, before quality enters.

Work backwards from 100 ms and subtract the costs fixed no matter which model you pick, debounce, the network hop if the model is on a server, and rendering:

100 ms budget - 40 ms debounce - 34 ms network round trip - 2 ms render  =  ~24 ms

So call it ~25 ms for decode, on the strictest assumption that the model is on a server. On the device the 34 ms of network disappears and roughly 58 ms is available. Either way, a candidate either fits in tens of milliseconds or it does not fit at all.

Why decode time is a division problem

Decode is limited not by arithmetic but by memory bandwidth: the bytes per second the accelerator can pull from its own memory. Each step reads every weight it uses and does little maths with each one, so it spends its time waiting on memory. That makes one decode step just bytes read over bytes per second:

step_time  =  (weight_bytes + batch × context × kv_per_token) / memory_bandwidth

The two things read every step are all the model’s weights, and the KV cache, the stored key and value vectors for every token already in context, which attention re-reads each step instead of recomputing. batch is how many users’ requests are served in one pass, which shares the cost of reading the weights.

The four candidates

Fix batch 64, 700-token contexts, and 3.3 TB/s of memory bandwidth (an H100-class accelerator, the reference part throughout). Two more terms: GQA (grouped-query attention) lets several query heads share one key/value head, so “GQA 4” means 4 KV heads and a proportionally smaller KV cache; int8/fp16 are the weight storage formats, at 1 and 2 bytes per weight. Each row is one division:

Nothing here is quoted from a benchmark. Each cell is two substitutions: kv/token = 2 × layers × kv_heads × head_dim × bytes (a key and a value), and the step-time formula above. For the 70B, weights of 140 GB plus a 14.7 GB KV cache read at 3.3 TB/s is 46.9 ms per step, so 281 ms for six tokens, the whole 100 ms budget spent 2.8 times over on decode alone, before a packet moves. The 7B’s 36 ms already exceeds the decode budget. Only the 300M fits.

Quality appears nowhere in that argument. The large models are excluded before it is discussed. The ordering is set by weight bytes, so changing the hardware moves every step time but not the ranking.

The model that fits

The survivor is a decoder-only transformer: it does exactly one thing, predict the next token from the tokens before it, with no separate module for reading an input.

Each layer holds attention (Q, K, V, O matrices; K and V are a quarter width because there are 4 KV heads to 16 query heads) and an FFN, the two-matrix feed-forward block that expands 1024→4096→1024 and does the per-token processing. The FFN is roughly three quarters of each layer. Adding it up: 24 layers × 11.0M plus 32.8M of tied embedding gives ~297M parameters, which is what “300M” rounds. Its KV cache is 2 × 24 × 4 × 64 × 2 bytes = 24 KB per token of context, the figure the step-time table used.

This model is trained from scratch on the mail corpus, not fine-tuned from a general model: the distribution is narrow, the vocabulary is domain-specific, and at 300M parameters capacity spent on general world knowledge is capacity not spent on this task.

Why decoder-only, not encoder-decoder

The single most important architectural decision comes down to one property of causal attention.

The obvious alternative is an encoder-decoder: an encoder reads the input and builds a representation, a decoder writes the output while consulting it. It is the natural shape for “prefix in, continuation out”, and it is wrong here. In an encoder-decoder the encoder is bidirectional, each token looks both backwards and forwards, so token i’s representation depends on tokens after i. Append one character and every source representation changes, so the entire encoder re-runs from scratch.

In a causal decoder, attention is masked so each token sees only itself and the tokens before it. Token i’s key and value depend on tokens 0..i and nothing after. Appending a token leaves every previous key and value bit-for-bit identical, which is exactly what makes them cacheable. The user extends the prefix on every keystroke, so this is the difference between re-encoding 700 tokens per keystroke and encoding 4.

flowchart TD
    subgraph DEC["Causal decoder — keystroke t+1"]
        A1["tokens 0..n<br/>K,V already cached, unchanged<br/>ZERO WORK"] --> A2["new tokens n+1..n+4<br/>prefill only these<br/>SMALL WORK"]
        A2 --> A3["decode 6 tokens"]
    end
    subgraph ENC["Bidirectional encoder — keystroke t+1"]
        B1["tokens 0..n+4<br/>every representation<br/>depends on every other"] --> B2["re-encode ALL 700<br/>nothing reusable<br/>ALL THE WORK"]
        B2 --> B3["decode 6 tokens"]
    end

The comparison is not “some work versus more work”. It is “zero plus small, versus all”, the 4-against-700 ratio. The decode 6 tokens step is identical on both sides: whatever you did to build the representations, writing the suggestion costs the same 6 decode steps, so the whole difference is attributable to the encoding strategy.

At fleet scale that ratio is the entire prefill budget. Prefilling one token costs about 2 × params operations (a multiply and an add per parameter), so at 500,000 requests/s:

re-encode 700 tokens each:   ~2.1e17 FLOP/s   ->  ~700 accelerators
prefill 4 new tokens each:   ~1.2e15 FLOP/s   ->  ~4 accelerators   (175x fewer)

The 175x is just 700 / 4 and survives whatever hardware you plug in.

One caveat: this holds while context is around 700 tokens. At 4,000 tokens the KV term stops being small next to the weights: for the 300M, a 4,000-token context needs ~6.3 GB of KV against 0.3 GB of weights. That is why the long-thread case is handled by a separate server tier on a relaxed budget.

The quality tradeoff is smaller than it looks

A 300M model is much worse than a 7B in aggregate, and that is mostly irrelevant, because aggregate quality is the wrong measurement for a system that can stay silent.

Measure top-1 agreement: how often the highest-probability next token from the 300M model matches the 7B’s (the 7B acting as a teacher, the reference the smaller student is judged against). Measure it over all positions, then over only the positions where the small model is confident:

The trigger only fires in the second regime. So the 29 points of aggregate disagreement collapse to 4 points on the suggestions actually shipped. Selective prediction, answering only when confident, is what makes a small model defensible: the model is allowed to be bad, as long as it knows when.

Distillation recovers part of the remaining gap for free. Instead of training the student only on which token actually came next, train it to match the teacher’s full probability distribution over its top-k candidates at each position (the teacher’s logits, the raw scores before normalisation). A sampled token is one fact per example; a distribution is what the teacher thought of every plausible alternative.

Training

Training cost follows a standard rule: a full step (forward, backward, update) costs about 6 × params × tokens operations, three times a forward pass alone.

6 × 3e8 params × 8e9 tokens  =  1.44e19 FLOP
÷ 8 accelerators at 300 TFLOP/s  ->  1.67 hours  ->  13.3 GPU-hours  ->  ~$33

On the other token reading (25.5B) the same arithmetic gives ~$106. The conclusion is identical either way: the model is not the expensive part. The data pipeline, the canary suite, the counterfactual replay harness, and the trigger calibration are the work. This ratio holds because the distribution is narrow: 8B tokens over 300M parameters is ~27 tokens/parameter, comfortable for a domain-specific model and thin for a general one.

Three things go on top of the base run:

• Int8 post-training quantization is the highest-return change in the project. Storing each weight as an 8-bit integer instead of a 16-bit float, after training, not during, halves the weight bytes: 600 MB → 300 MB, +0.6% perplexity. Because decode is memory-bound, the device decode step halves almost exactly, 8.8 ms → 4.4 ms. (The server step falls by less, 0.52 → 0.42 ms, because the KV cache did not shrink and is a large share of the bytes read at batch 64.)
• Per-language models, trained separately, for the capacity reason above: at 300M parameters one model per language beats one multilingual model of the same size.
• On-device personalization is a small adapter, a thin set of extra weights trained on the frozen base model, trained on the individual’s own sent mail, on their device, never uploaded. This is where acceptance rate actually moves, because people have idiosyncratic sign-offs, and it has no privacy exposure at all.

The trigger policy

The model decides what to say; the trigger decides whether to say it and how long to make it. Because the two error types cost wildly different amounts, that decision has an exact answer, not a tuned one.

The cost asymmetry, priced

The four things that can happen once a suggestion is shown, normalised into one unit, the utile, anchored so that 1.00 utile is one average accepted suggestion (18 characters saved, ~5 seconds of typing). Throughout, q is the calibrated probability the suggestion is correct.

Five symbols come out of this and drive the rest: A_ok = 0.55 (probability a correct suggestion is accepted, well under 1, because users often keep typing), A_wrong = 0.03 (probability a wrong one is accepted anyway, on Tab autopilot), V_a = 1.00 (value of an accept), C_r = 0.05 (cost of a shown-and-dismissed suggestion), and C_w = 30 (cost of a wrong suggestion accepted and sent). The last row is 30x anything else in the table, and the whole design is built around it.

Show the suggestion when its expected value is positive:

q · A_ok · V_a   >   (1 - q) · (C_r + A_wrong · C_w)

Substituting and solving gives the threshold:

C_fp = C_r + A_wrong · C_w = 0.05 + 0.90 = 0.95      (cost of showing a wrong one)
C_fn = A_ok · V_a          = 0.55                    (value forgone by staying silent)
q*   = C_fp / (C_fp + C_fn) = 0.95 / 1.50 = 0.633

Show the suggestion when the calibrated probability of being right is at least 0.633. That is exactly the standard cost-matrix threshold C_fp / (C_fp + C_fn) (choosing a threshold from the cost matrix), where C_fp is the cost of a false positive (showing a wrong suggestion) and C_fn the cost of a false negative (staying silent on a good one).

Compare it against the two thresholds you might guess:

The gap between 0.083 and 0.633, a factor of 7.6, is entirely the A_wrong · C_w term, a 3% chance of a bad accept priced at 30 times a good one. Reasoning only about typing time lands almost an order of magnitude low, because every extra suggestion a low threshold buys comes from the low-q region, exactly where the bad-accept term dominates. The failure then shows up as feature-disable rate, not on the quality dashboard, which is why it can run for a quarter unnoticed.

The design is more sensitive to A_wrong than to any hardware number: at 0.005 (users who always read) the threshold drops to 0.27 and coverage roughly doubles; at 0.10 (heavy autopilot) it rises to 0.85 and the feature nearly stops firing. Both A_wrong and C_w are measurable online, so the first launch is deliberately conservative and exists mainly to measure them.

The calibrator

The number fed to that threshold cannot be the model’s own sequence probability (the product of its per-token probabilities), which fails twice:

• It is uncalibrated. Among suggestions the model scores 0.7, the fraction correct is not 70% (calibration).
• It is length-confounded. Being a product of numbers below 1, it shrinks with every token regardless of quality, so a fixed threshold silently caps suggestion length. At a good mean per-token probability of 0.92, a 0.633 cut lets 0.92^5 = 0.659 through but blocks 0.92^6 = 0.606, capping suggestions at 5 tokens for reasons unrelated to whether long suggestions are good. Since the optimal length below is 5-6 tokens, that truncates the answer exactly where it matters.

The fix is a second, tiny model, a calibrator, trained on held-out data to predict the one thing the threshold needs: P(the user's actual continuation starts with this suggestion). It must be free at inference, a logistic regression, or a shallow GBDT (gradient-boosted decision tree, an ensemble where each tree corrects the previous one’s errors). Its inputs are features of the model’s output, not the text:

Two of these are worth defining. Margin is the gap between the best and second-best continuations under beam search (a decoder that keeps several partial continuations alive at once instead of committing to one token per step); a large margin means one obvious continuation. Entropy measures how spread out a distribution is (near zero when one token dominates, large when many are plausible) and entropy at the first token predicts failure best because that is where the model commits. Threshold the calibrated q at 0.633 and coverage lands near 12-14%.

Length is the same decision, not a separate one

Choose the length L that maximises expected utility. The value of an accept grows with length (at 4.2 chars/token, an L-token suggestion saves 4.2L/18 utiles), while the probability of being right shrinks with length. Let q(L) be the running product of per-position accuracies, and 0.95 be C_fp from above:

U(L)  =  q(L) · A_ok · (4.2L / 18)  -  (1 - q(L)) · 0.95

Run this on two populations, because the difference between them is the whole point:

Over all contexts, the optimum is 2 tokens and anything past three is net harmful. That is the unconditional optimum, the best length averaged over every position. But the trigger never fires on an average position; it fires on the confident 12%, whose per-position accuracies are much higher (0.99, 0.98, 0.97... versus 0.94, 0.91, 0.88...). On that slice the optimum moves to 5-6 tokens and peak utility rises from +0.082 to +0.402, a factor of ~5. That entire difference is the value of the trigger: not a better model, but permission to be more ambitious where you are already right.

The whole policy is these two functions. should_suggest prices one candidate; best_length sweeps L, accumulating q(L) as the running product. The threshold is computed inside, not passed in: the point of the derivation is that it is not a tunable.

def should_suggest(q: float, length_tokens: int, *, a_ok=0.55, a_wrong=0.03,
                   c_read=0.05, c_wrong=30.0, chars_per_token=4.2,
                   chars_per_utile=18.0) -> dict:
    """Expected-utility trigger. q must be CALIBRATED, not a raw sequence prob."""
    c_fn = a_ok                            # value forgone by not showing
    c_fp = c_read + a_wrong * c_wrong      # cost of showing a wrong one
    threshold = c_fp / (c_fp + c_fn)
    value = chars_per_token * length_tokens / chars_per_utile
    utility = q * a_ok * value - (1 - q) * c_fp
    return {"show": q >= threshold and utility > 0,
            "threshold": threshold, "utility": utility}


def best_length(per_position_acc, **kw):
    """Pick L jointly with the show/no-show decision. Returns (L, utility)."""
    best, q = (0, 0.0), 1.0
    for length, acc in enumerate(per_position_acc, start=1):
        q *= acc
        u = should_suggest(q, length, **kw)["utility"]
        if u > best[1]:
            best = (length, u)
    return best

Both length tables must be re-derived per language, because a tokenizer that fragments a script produces a completely different chars-per-token figure, the failure traced under failure modes.

Hard suppression, before the trigger

Some classes are never worth their expected value at any confidence the system can certify, so they are excluded in code before the trigger runs. Every row is enforcement, not advisory:

A protected attribute is a characteristic that law and policy forbid making decisions on: race, religion, disability, sexual orientation, national origin. The proximity rule suppresses near any such term because the corpus contains real-world associations a completion would faithfully reproduce.

The gendered-pronoun rule falls straight out of the same utility formula, which is why it belongs here and not in a values statement. Two inputs change: a pronoun saves ~3 characters, so V_a = 3/18 = 0.17; and a pronoun error is a personal misattribution about a named colleague in a sent message, so price it at C_w = 400. The same cost-matrix ratio then gives:

C_fp = 0.05 + 0.03 × 400 = 12.05      C_fn = 0.55 × 0.17 = 0.094
q*   = 12.05 / 12.14 = 0.992

A tiny payoff over a large risk pushes the required confidence to 99.2%. A calibrated 99.2% on a demographic inference the model draws from names and job titles is not attainable, and would not be trustworthy if the number claimed it was. When the derived threshold exceeds what calibration can certify, suppress the class entirely, a real design decision recovered from arithmetic, not adopted from a headline.

Serving

Follow one keystroke through the running system. It is easiest to read as four gates: two that drop work before the model, the model, and two that drop work after.

Before the model, each keystroke increments a sequence number (a counter used later to tell whether an arriving result still describes the text on screen), then meets the debounce gate, 40 ms of quiet, at a word boundary. A keystroke that fails it is dropped with no request issued. Survivors pass the hard suppression rules; a blocked one is dropped there. Both gates cost almost nothing and are placed first for that reason.

At the model, what is left reaches the on-device 300M int8 model, whose KV cache is already warm for this composition, and its output goes to the calibrator, which produces q. The model must decode all 6 tokens before either remaining check can run, which is why the decode budget was fixed first.

After the model, the trigger asks whether q clears 0.633 and the chosen length’s utility is positive; if not, show nothing. If it passes, one final check confirms the sequence number is still current. If not, the result describes text already typed past, so discard it. Only then does the client render grey inline text. The dotted path is the one exception: for the first suggestion in a long reply thread, a server tier holding the full thread context on a relaxed 400 ms budget produces the candidate, rejoining at the same calibrator.

flowchart TD
    K([Keystroke]) --> SEQ["Increment sequence no."]
    SEQ --> DB{"Debounce:<br/>40 ms quiet AND<br/>at a word boundary?"}
    DB -->|no| DROP1["Drop. No request."]
    DB -->|yes| SUP{"Hard suppression<br/>rules"}
    SUP -->|blocked| DROP2["Drop"]
    SUP -->|clear| ODM["On-device model<br/>300M int8<br/>KV cache warm"]
    ODM --> CAL["Calibrator -> q"]
    CAL --> TRG{"q >= 0.633<br/>and U(L) > 0?"}
    TRG -->|no| DROP3["Show nothing"]
    TRG -->|yes| FRESH{"seq no. still<br/>current?"}
    FRESH -->|no| DROP4["Stale. Discard."]
    FRESH -->|yes| SHOW([Render grey inline])
    ODM -.->|first suggestion in a<br/>long thread only| SRV["Server tier<br/>full thread context<br/>relaxed 400 ms budget"]
    SRV -.-> CAL

KV cache reuse as the user types

Every keystroke produces a prefix that extends the previous one. Because attention is causal, the keys and values of tokens 0..n are unchanged by appending token n+1, so the cache from the previous suggestion is still valid and prefill runs only over the new suffix.

flowchart LR
    T1["t=0<br/>'Thanks for sending'<br/>prefill 4 tokens"] --> C1[("KV cache<br/>4 tokens")]
    C1 --> T2["t=1<br/>+' that over. I'll'<br/>prefill 4 NEW tokens"]
    T2 --> C2[("KV cache<br/>8 tokens")]
    C2 --> T3["t=2<br/>+' review'<br/>prefill 1 NEW token"]
    T3 --> C3[("KV cache<br/>9 tokens")]
    C3 --> INV["Backspace past token 6"]
    INV --> TRUNC["Truncate cache to 6.<br/>Only tokens 7+ invalid"]

Editing is handled by the same rule, not a special one: a backspace or cursor move invalidates the cache from the edit point forward and only from there, because everything before the edit still depends only on tokens before the edit. Truncate to the longest common prefix and re-prefill the rest.

This is a throughput argument, not a latency one. Prefilling 700 tokens on a 300M model costs about 1.4 ms, noise against 100 ms. What the cache saves is ~696 accelerators of fleet-wide prefill (the 175x above). On the device, where compute is roughly 200x scarcer than on a server accelerator, it also matters for latency.

The cache is why the model runs on the device

The deployment decision comes from a memory calculation, not the privacy argument you might expect. Server-side caching hits a memory wall:

340-char message at 3.3 chars/s          =  103 s of composing per session
50M × 6 composes × 103 s / 86,400 s      =  358k concurrent, ~1.07M at peak
KV per session  =  700 tokens × 24 KB    =  17.2 MB
1.07M sessions × 17.2 MB                  =  18.5 TB  ->  231 accelerators, cache only
decode: 500k peak QPS × 6 tokens         =  ~20 accelerators

Cache residency costs ~11x more hardware than the computation: 231 accelerators that do no arithmetic, they only hold bytes, against 20 that decode. Three ways out: evict caches on a short inactivity TTL (time-to-live) and re-prefill on return; page the cache to host memory over PCIe (~0.27 ms for 17.2 MB, affordable); or put the model where the cache is naturally free. That third option is the design: on the device there is exactly one session, 17.2 MB is nothing, and it lives next to the only user allowed to see it. The memory arithmetic, not privacy, forces on-device deployment; the privacy benefit arrives second.

On-device versus server

Compared end to end, the two deployments are separated by the tail, not the median:

ON DEVICE                        SERVER
debounce           40.0 ms       debounce            40.0 ms
prefill 4 tokens    1.5 ms       client -> edge      18.0 ms
decode 6 @ 4.41    26.5 ms       queue + admission    5.0 ms
render              2.0 ms       prefill 4 tokens     0.2 ms
                  --------       decode 6 @ 0.42      2.5 ms
      p50 =        70.0 ms       edge -> client      16.0 ms
      p99 =        78   ms       render               2.0 ms
      (thermal throttling)                          --------
                                       p50 =         83.7 ms
                                       p99 =        ~210 ms
                                       (network tail)

The p50 difference is ~14 ms and would decide nothing; both medians sit under 100 ms. The p99 difference is 132 ms, and p99 is the budget, because the suggestion that misses the deadline is the one that renders after the user has typed past it. The two tails have different causes and different fixes: the device’s p99 is set by thermal throttling (the phone slowing when hot), which is bounded and yours to manage; the server’s is the network tail, which no amount of server capacity fixes because the delay is not on your hardware.

Once on the device, the phone’s memory bandwidth caps the model size. Phone memory runs at ~68 GB/s (LPDDR5X) against a server’s 3.3 TB/s, about 48x less. Since decode is bandwidth-bound, a 300M int8 model is 4.41 ms/token (26.5 ms for six (fits), while a 1B model is 14.7 ms/token (88.2 ms for six) does not fit, since the budget also owes debounce and render). The memory bus, not the accuracy curve, caps the model at a few hundred million parameters.

The server tier still earns its place for one case: the first suggestion after opening a long reply thread, where 4,000 tokens of context materially help. The user has just clicked reply and is not yet typing, so the deadline there is 400 ms, the latency assumption relaxing in the one moment it genuinely does not apply.

The fleet is not uniform, though. A five-year-old handset has a fraction of the bandwidth, turning a 26.5 ms decode into 80 ms and blowing the p99 on precisely the devices hardest to spot on a fleet dashboard. The fix: segment the latency guardrail by device class, and below the bar fall back to the n-gram model, which is microseconds everywhere.

Debouncing and cancellation

Firing on every keystroke is wasteful and worse for the user. Debouncing waits for a short pause before acting, so a burst of fast keystrokes produces one request instead of ten:

That last row is where the “48 evaluations per compose” figure comes from: at 14 per 100 keystrokes, a 340-character message produces ~48 evaluations, and every traffic number downstream follows from it.

The 40 ms threshold looks pointless against the mean and is not. At 6.7 chars/s the average gap is 150 ms, and a 40 ms gate on 150 ms gaps would pass everything. But the table shows it cuts word-boundary requests from 22 to 14, meaning 36% of word-boundary gaps are shorter than 40 ms, which a 150 ms mean cannot produce. Only burstiness can. Real typing is runs of practised muscle memory (a familiar word, a sign-off typed ten thousand times), where successive keystrokes land 20-30 ms apart, separated by genuine pauses for thought of several hundred ms. The distribution is bimodal and the 150 ms mean sits in the empty trough between the two modes. The gate is placed inside that trough (above the intra-burst mode, below the inter-burst one) so it separates the two populations instead of trimming a tail. The 36% it removes are mid-burst keystrokes where the user is executing a word already decided on; the 64% it passes are the pauses where a completion is worth reading. Derive the number from your own inter-keystroke histogram, never from the mean.

Cancellation is not optional. Every keystroke bumps the sequence number; a result carrying a stale sequence number is discarded on the client, and any in-flight server request is cancelled. Without this you ship the worst bug the feature has, a suggestion fading in after the user has typed past it, forcing them to re-read text they already wrote.

Metrics

Some of this can be measured before launch, some only after, and the metric everyone reaches for first, acceptance rate, is the one that will mislead you.

Offline

Computed on held-out sent mail before anything ships. Only one is a headline; the rest are gates and diagnostics.

Exact match is the right metric here, which is unusual for a generative system. Generative outputs usually have many acceptable forms, making exact comparison useless. Here the acceptable set has size one: the user gets a single Tab, so a suggestion is useful only if it is a literal prefix of what they were going to type. A paraphrase scores zero and deserves zero. BLEU and ROUGE (overlap-based text metrics that score by shared n-grams) are actively wrong: a 6-token span has no room for n-gram overlap to mean anything, and there is no “70% accept” key for partial credit to cash.

The counterfactual replay is the offline harness that comes closest to the online number. (A counterfactual measurement asks what would have happened under a policy that was never run.) Take held-out sent messages, walk every position, run the real trigger and length policy at each, and count characters that would have been saved had the user accepted every suggestion that exactly prefixes what they typed. It is the only offline number that tracks the online one, because it exercises the decision layer, not just the language model. It is also systematically optimistic, and the size of the optimism is a parameter you already estimated:

simulated chars/day  566    <- assumes every exactly-correct suggestion is accepted
observed  chars/day  316
ratio               0.56    =  A_ok, the measured P(accept | correct)

So a change in the sim-versus-reality gap signals that user behaviour moved, not that the simulator broke. Calibrate the replay once against a live A/B, then use it as a gate, never as a forecast.

Online

The headline is characters saved per user per day, and every input has already been derived:

evaluations/user/day  =  6 × 48                          =  288
suggestions shown     =  288 × 0.14 coverage             =  40.3
acceptance            =  0.78 × 0.55 + 0.22 × 0.03       =  0.436
accepts/day           =  40.3 × 0.436                    =  17.6
chars saved/day       =  17.6 × 18                       =  316   (15.5% of typing, ~95 s)

The acceptance line is the only one that is not a plain multiplication: of the shown suggestions, 0.78 are correct and accepted 55% of the time, and 0.22 are wrong and accepted 3% anyway. Acceptance rate is derived here, not measured and set. If your reported acceptance rate does not reconcile with your threshold, calibrator, and behavioural rates, one of the three is wrong, and that reconciliation failure is the most valuable signal the system produces.

Now price the same day in utiles, charging for the suggestions that went wrong:

gain        17.6 accepts × 1.00              =  +17.6
dismissals  22.7 dismissed × 0.05            =   -1.1
bad accepts 0.27/day accepted-wrong × 30     =   -8.0
                                                ------
                                                 +8.5

Bad accepts consume 45% of the gross value of the feature (8.0 / 17.6), at a rate of about one every four days per user. That single line is the whole argument for a precision-first design, and it is why the threshold lands at 0.633, not 0.083. The threshold derivation and this product accounting were done independently and agree, which is the main reason to trust either.

Why acceptance rate misleads

Acceptance rate (accepts / suggestions shown) has the shape of CTR (click-through rate). It fails as a headline for three reasons:

1. You control the denominator. Raising the threshold shows fewer, better suggestions, so acceptance rises while the product gets worse:

   Threshold	Coverage	Acceptance	Chars saved/day
   0.633	14.0%	43.6%	316
   0.80	6.0%	49.3%	153
   0.90	2.5%	52.4%	68

   Acceptance improves 20% while characters saved falls 79%. Any metric whose denominator is a config knob is a metric you will tune.

2. There is no impression the user opted into. A suggestion appears unbidden mid-sentence; a non-click is not disinterest. It is often cost, a fixation spent on wrong text.

3. A rejection has a negative price, and CTR counts it as zero. That is the C_r term. A click-through framing recommends showing far more than the utility calculation permits.

Report the pair, characters saved and suggestions shown, or the single utility number that prices both.

Guardrails and the A/B test

A guardrail does not have to improve for a launch, but its regression blocks the launch regardless of the headline. The five that catch what characters-saved cannot see:

Retraction rate is the one to instrument first: acceptance says the user pressed Tab; retraction says whether they meant it, and it is the only direct read on the A_wrong the threshold hinges on.

The A/B test randomizes on the user, never the request: habituation, trust, and the decision to disable accumulate per person, and randomizing per request understates variance and manufactures results that do not replicate (A/B testing). Sizing with n ≈ 16 σ² / δ² to detect a 4% change in chars saved (δ = 12.6) with a heavy right tail (σ = 350) gives ~12,300 users per arm, ~7,400 with CUPED (a variance-reduction technique that subtracts each user’s own pre-experiment behaviour, here typing volume). Both are trivial against 50M DAU, so the binding constraint is time, not users: read the experiment at week 3, not week 1, because a new inline suggestion gets tried before it gets used, and the week-1 number measures novelty. Fix the read window before the experiment starts: the whole cost model assumes A_ok and A_wrong are properties of users, and the week-1-vs-week-3 gap is direct evidence they are not, early in a user’s exposure.

Scale and cost

Every dollar figure is the request rate times a unit price:

50M DAU × 6 composes × 48 evaluations  =  14.4B requests/day
                                          167,000 QPS average, ~500,000 peak (3x factor)

Priced against that load:

(MTok is a million tokens, billed as input/output; NPU is the phone’s dedicated accelerator.) The API bill is easy to disbelieve, so it is worth seeing where the money goes: each request sends 700 tokens in and gets 6 out, and the input is 96% of the bill. You pay to re-send the same growing prefix 48 times per compose, the exact work the on-device KV cache does for free. The API path is disqualified twice: first by a 36 ms decode step that blows the budget before the network is involved, then by a bill three orders of magnitude above the alternative. The latency argument is the one that cannot be engineered around.

Battery is the real cost of the on-device design, and it is small: 288 inferences/day × ~28 ms of NPU each = ~8.1 seconds of NPU per day, negligible against screen-on time. A 1B model at 88 ms per inference would be ~25 seconds a day and a measurably warm phone, the same memory-bandwidth constraint that capped the model size, arriving again in a different currency.

The deeper reason on-device wins: its bill is $0 marginal at any traffic level. It is the only option whose cost does not scale with success.

Failure modes

Seven traces of the system going wrong, each with its mechanism and the specific control that stops it.

1. A factually wrong completion. Given confirming the invoice total is $, the model completes 1,200.00 and payment is due within 30 days at p = 0.71, high confidence, learned from thousands of near-identical invoices, and completely uninformative because nothing in the model can know the real total. Guard: hard suppression inside numeric, currency, date, and time tokens. Detection: retraction rate spikes on messages containing currency tokens.

2. A biased completion. Given I met with the cardiologist this morning and I'll ask , the model completes him..., inferring gender from a job title because corpus co-occurrence says so. Guard: the pronoun suppression derived above (q* = 0.992, unattainable). This is not a special case bolted on; it is the same utility rule with a different C_w. The related trace, a protected-attribute term near the cursor turning the suggestion into a stereotype, gets the same treatment by the proximity rule.

3. Memorized personal data from another user. Given the routing number is , the model reproduces a real routing number it saw in training. Detection: the canary suite. Guards, layered: DP bounds any single user’s influence; the k-anonymity whitelist blocks any string not seen across 50+ users; the numeric suppression rule blocks the shape regardless.

4. The stale suggestion race. A request fires at seq=41; the user types another character (seq=42) before the seq=41 response arrives 12 ms later; without a sequence check it renders, offset by one character, describing a prefix that no longer exists. Guard: compare the sequence number on arrival, discard mismatches, cancel in-flight requests on keystroke. A five-line fix whose absence is the most common way the feature feels broken.

5. Cross-session cache reuse. Keying the cache on sha256(prefix_tokens) alone means two users composing Hi, collide, and on a longer prefix the collision leaks one user’s content into another’s suggestion. (Email openings are formulaic, so collisions are constant.) Guard: key on (session_id, prefix_hash) and scope every entry to its session. On the device this cannot occur at all, a third independent argument for on-device.

6. Non-Latin scripts degrade silently. The tokenizer fragments non-Latin scripts far more aggressively than English (tokens), so a 6-token suggestion carries 2 words instead of 5. Characters saved per suggestion halves while acceptance rate looks unchanged, so the aggregate dashboard shows nothing. This also affects code-switching (moving between two languages in one message), which is ordinary for much of the world. Guard: segment every metric by language and script, and set each language’s length policy from its own q(L) curve.

7. The feature eats its own training data. The slowest failure, over quarters: users accept suggestions, so sent mail gets more templated; the next training round sees a more templated corpus, so suggestions get blander; acceptance rises because bland suggestions are safe; but value per accept falls, because a suggestion the user would have written anyway saves nothing that matters. Every dashboard number improves while the product decays, the shape of unlabeled drift. Guard: hold out a permanent control cohort never shown suggestions, treat their sent mail as the reference distribution, and monitor the KL divergence (a measure of how far one distribution has moved from another) between the treatment and control cohorts’ sent-mail n-gram distributions. If it grows monotonically over quarters, the loop is closing.

Alternatives considered

Each design a reasonable person would propose instead, with the specific number that kills it. (seq2seq is the encoder-decoder framing above; KenLM is a widely used n-gram language model, next word purely from counts, microseconds per query, no generalisation beyond runs it has literally seen; a semantic cache returns a stored answer for any request similar to an old one, a win when generation is expensive and a liability when it is cheap.)

What the design rests on

Three assumptions carry the whole thing; if any is false the design is not suboptimal, it is invalid.

• The p99 budget is 100 ms, keystroke to pixels, and it is set by fast typists (a 150 ms gap), not average ones. This decides model size, architecture, deployment, and debounce. At a 2-second budget this becomes an ordinary hosted-model product and most of this design is unnecessary; at a real 300 ms gap, a 7B on a server fits and on-device becomes a preference.
• A_wrong = 0.03: people accept wrong suggestions about 3% of the time. This is the single most sensitive number, moving the threshold from 0.083 to 0.633 via C_fp. It is currently an estimate, and it is the first thing the conservative launch exists to measure.
• The user base is in the tens of millions. Differential privacy’s noise falls as 1/m, and the k-anonymity whitelist needs 50 distinct users per phrase. At 200,000 users this is a different design: no federated training, no whitelist, a filtered public corpus, and much broader suppression.

Two more that hold up large parts of the design: confidence is informative, so selective prediction works and a 300M model is defensible; and sent mail is the right distribution to imitate, which the whole data pipeline enforces (measure per-position accuracy on live prefixes, never held-out perplexity, to catch drift). The remaining numbers (50M DAU, 6 composes/day, 340 chars, batch 64, 700-token contexts, H100-class hardware, 4.2 chars/token) you are free to pick; being wrong about them costs a re-derivation, not a redesign.

Conclusion

• Size the model from the latency budget, not a benchmark. A fast typist’s 100 ms inter-keystroke gap allows ~25 ms of decode, which admits a 300M model (2.5 ms) and excludes a 7B (36 ms) and a 70B (281 ms) before quality is ever discussed. A suggestion conditioned on a stale prefix is worse than silence.
• Decoder-only, because causal K,V survive an append. The user extends the prefix every keystroke; a bidirectional encoder would re-encode all 700 tokens each time (175x the prefill), while a causal decoder prefills only the new tokens and reuses the cache.
• A small model is good enough because it gets to stay silent. The trigger fires only on the confident 12% of positions, where the 300M agrees with a 7B 96% of the time. Selective prediction, not model size, carries the quality, and it lets suggestions grow from 2 tokens to 5-6 on the fired slice.
• The threshold is derived, not tuned. q* = C_fp / (C_fp + C_fn) = 0.633, where the bad-accept term (A_wrong · C_w) moves it from 0.083 to 0.633. Feed it a calibrated probability, because the raw sequence probability is length-confounded.
• On-device because KV cache residency costs ~11x more server hardware than decode, and the server p99 is 210 ms against a 100 ms budget. Privacy is the second reason, not the first.
• Measure characters saved, not acceptance rate, whose denominator you control. Watch retraction rate as the free read on A_wrong, and remember that bad accepts eat 45% of the gross value, which is what precision-first design is protecting.
• Privacy is the data architecture: never log the prefix, federated learning with differential privacy (measured by canary exposure), and a k-anonymity whitelist on the output. Each has a measured utility cost; a mechanism quoted without one is a slogan.

One line to remember: this is a latency-and-precision problem wearing a text-quality costume, so let the 100 ms budget pick the model and the cost asymmetry pick the threshold.

Further reading

• Chen et al., “Gmail Smart Compose: Real-Time Assisted Writing” (KDD 2019), Google’s production system.
• McMahan et al., “Communication-Efficient Learning of Deep Networks from Decentralized Data” (2017), federated learning.
• Abadi et al., “Deep Learning with Differential Privacy” (2016), DP-SGD (clipping plus Gaussian noise).
• Mironov, “Rényi Differential Privacy” (2017), the accountant behind the epsilon figure.
• Carlini et al., “The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks” (2019), canary insertion and exposure.
• Hinton et al., “Distilling the Knowledge in a Neural Network” (2015), training a small student from a large teacher.
• Geifman & El-Yaniv, “Selective Classification for Deep Neural Networks” (2017), answering only when confident.
• Deng et al., “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)” (WSDM 2013), the variance reduction used in the A/B sizing.

Next: Machine Translation, where the output space is narrow enough that reference-based metrics come back to life, and the hard problem moves to the tail of the language distribution.