02 — Smart Compose

“Build inline sentence completion for our email client. As the user types, suggest the rest of the sentence in grey; Tab accepts.”

Smart Compose is the feature that finishes your sentence while you type an email: you have written half a sentence, grey text appears proposing the rest, and pressing the Tab key accepts it.

The whole system falls out of two numbers:

1. How long you have before the user’s next keystroke.
2. How much more a wrong suggestion costs than showing nothing at all.

By the end of the chapter you will be able to do four things. Size a model from a latency budget instead of from a benchmark. Derive the confidence threshold that decides whether to show a suggestion at all. Explain why this feature runs on the user’s phone rather than in a data centre. And name the assumptions the whole design would collapse without.

The sentence to open the interview with: “This is not a text-quality problem, it is a latency and precision problem. The budget is one gap between keystrokes — call it 100 ms at the 99th percentile — and that disqualifies every model I would otherwise reach for, before we have discussed quality at all. And a wrong suggestion costs strictly more than no suggestion, so the threshold that decides when to show one is derived from that asymmetry, not chosen.”

Terms used throughout

Six terms appear on nearly every page below. Pin them down now so nothing later depends on a word you are still guessing at.

One more term deserves a paragraph rather than a table row, because the chapter is organised around it.

A latency budget is the total time the system is allowed to take. You fix it before anything is designed, and then you spend it: every stage of the pipeline draws from the same pot, and when the pot is empty the design is over.

That is what “size a model from a latency budget instead of from a benchmark” means. You do not pick a model and then measure how fast it is. You decide what you can afford, and let the budget eliminate models before quality is ever discussed.

Prompt caching matters here for one specific reason: the cached state for a given token depends only on the tokens before it. That single property is what this chapter’s entire architecture is chosen to preserve, and the decoder-only argument is nothing but that property spelled out.

This is the first and cleanest application of the framework in chapter 01, because one framing answer decides almost everything else. It does not require that chapter, though: everything borrowed from it is restated here.

Problem framing

Start with exactly what goes into the system and what comes out, and the traffic and latency numbers the rest of the chapter will spend.

Say the input and the output first

Say the input and the output as one sentence before touching the mechanism. The shape of both constrains everything after.

The half-written sentence goes in — the characters typed so far, called the prefix, together with the subject line, the recipient’s domain and the thread being replied to — and 2 to 8 tokens of proposed continuation come out.

The most important thing about that output is that nothing is also a valid output. In fact it is the output most of the time. The suggestion is shown on only 12-14% of the moments where the system is consulted (that figure is derived later); on the other 86-88%, the system deliberately shows the user nothing at all.

A design that treats “produce a continuation” as the task has already missed the product.

Which framing question actually decides this design

Chapter 01 opens a design round with five questions whose answers change the architecture rather than just the arithmetic (ch 01 stage 1):

1. What is the latency budget at p99?
2. What does a wrong output cost?
3. Who reviews the output, and when?
4. What is the ceiling on cost per request?
5. Is the output user-facing, or does it feed another system?

Here the answers are unusually lopsided, which is what makes this the cleanest example in the folder.

• Question 1 overrides everything else. The budget is so tight that it eliminates most models before quality is mentioned.
• Question 2 sets the operating point. The asymmetry between a wrong suggestion and no suggestion is what the trigger threshold is computed from.
• Questions 4 and 5 barely matter. The output is user-facing prose with no schema for a downstream parser to reject, and every option that survives question 1 meets the cost ceiling with room to spare.

Assumptions in this stage.

State out loud: 50M daily active users, 6 composes a day of about 340 characters each, and 4 characters per token. Being wrong about these costs a re-derivation of the fleet size, not a redesign.

Load-bearing: the 100 ms p99 budget, and the claim that a wrong suggestion costs more than no suggestion.

If the budget were 2 seconds — a chat product rather than an inline one — a hosted frontier model over the network becomes legal, and the entire model-selection argument below evaporates. If wrong suggestions were free, the threshold collapses towards showing everything, and the trigger layer, which is most of this chapter, stops existing. Everything after this point is downstream of those two sentences.

Where the 100 ms budget comes from

The number the rest of the chapter is built on deserves a derivation rather than an assertion, and the derivation is human typing speed.

The 100 ms is not a round number someone liked. It is the interval between one keystroke and the next — the inter-keystroke interval — for a user typing quickly.

Typing speed is conventionally quoted in wpm, words per minute, where a “word” is standardised at five characters. Converting wpm to a gap between keystrokes takes three steps:

1. Multiply by 5 to get characters per minute.
2. Divide by 60 to get characters per second.
3. Take the reciprocal to get seconds per character — which is the gap between one key and the next.

Run that on an average typist and a fast one:

average typist   40 wpm  =  200 chars/min  =  3.3 chars/s  ->  300 ms between keys
fast typist      80 wpm  =  400 chars/min  =  6.7 chars/s  ->  150 ms between keys

Worked out for the fast typist: 80 x 5 = 400 chars/min, 400 / 60 = 6.7 chars/s, 1 / 6.7 = 0.150 s. Both the 3.3 chars/s and the 150 ms come back several times later, so they are worth remembering.

A suggestion that arrives after the next keystroke is not late — it is wrong, because the prefix it was conditioned on no longer exists. It renders, the user’s eye moves to it, and it disappears on the very next character. That is strictly worse than showing nothing: it cost the user a fixation — one deliberate movement of the eye onto the grey text, which takes a fraction of a second and cannot be taken back — and delivered no value in return.

So the budget is the fast typist’s interval with margin for the work that is not the model: 100 ms at p99, measured from the keystroke to the pixels appearing on screen.

This assumption is the one the design lives or dies on, so state it as such. Every subsequent decision — the 300M-parameter model, the decoder-only architecture, the on-device deployment, the debounce policy — is a consequence of 100 ms and of nothing else.

Three ways it could be false, and what each one breaks:

The third row is the one worth doing the arithmetic on, because “1 in 100” sounds either alarming or harmless depending on what you multiply it by. Multiply it by the traffic numbers already stated — 48 model evaluations per compose, 6 composes a day, and a 14% chance that any given evaluation renders anything at all:

misses per compose        48 x 0.01              =  0.48    (one every ~2 messages)
misses per user-day       288 x 0.01             =  2.9
user-VISIBLE late renders 0.48 x 0.14            =  0.067   (one every ~15 messages)

Only the third line is what the user experiences, because a miss on an evaluation that was going to show nothing anyway is invisible. So the real rate is one bad render every fifteen messages.

That is not “several per message”. It is a slow drip — and the drip is the point. It is frequent enough that a heavy composer meets it weekly, and rare enough that no dashboard will surface it.

What interviewers probe: whether the latency number is derived or asserted. “Users expect it to feel instant” is a weak answer. “The budget is the inter-keystroke interval of a fast typist, because a suggestion conditioned on a stale prefix is worse than silence” is the answer.

ML objective

The system has to learn two things: what to say, and whether to say it. The second is where the product actually lives, and conflating the two is the most common design error in this problem. An objective in this sense is the quantity training tries to make as small or as large as possible; the design error is assuming there is only one.

The generation objective is the standard one for a language model: predict the next token, and penalise the model in proportion to how surprised it was by the token that actually came next. That penalty is called cross-entropy — the negative logarithm of the probability the model assigned to the true token, summed over the sequence — and minimising it means maximising the probability the model would have assigned to text the user really wrote. Here it runs over the user’s own sent mail, conditioned on (recipient, subject, thread, prefix):

L  =  - sum_t  log P(x_t | x_<t, context)

Read that as: the loss L is the sum, over every position t in the message, of minus the log-probability the model gave to the token x_t that actually appeared, given everything before it (x_<t) and the surrounding context. A confident correct prediction contributes almost nothing; a confident wrong one contributes a lot.

The decision objective is separate, and it is where the product lives. Given a candidate continuation and the model’s probabilities, decide two things: whether to show anything at all, and how long the suggestion should be.

That is a calibrated binary decision with an asymmetric cost matrix. Both terms carry weight:

• Calibrated means the number the system calls a probability really behaves like one. Among the cases it scores 0.7, roughly 70% turn out right.
• Asymmetric cost matrix means the two ways of being wrong — showing a bad suggestion, and withholding a good one — do not cost the same.

It is not a language-modelling problem, and no amount of language-modelling progress solves it.

Perplexity is a gate, not a goal. Perplexity is the standard summary of a language model’s quality: the exponential of the average cross-entropy. It is interpretable as the number of equally-likely options the model was effectively choosing between at each token, so lower is better.

A model with 8% lower perplexity that fires the trigger no more often ships zero additional characters saved. The product metric is characters saved per user per day at a fixed annoyance budget, and that number is the product of the language model and the decision layer — not of the language model alone.

Assumptions in this stage.

Load-bearing: that the decision layer can be built at all. Concretely, that a usable calibrated confidence can be extracted from a 300M model’s own outputs.

Suppose it could not — suppose the model is no more often right when it claims 0.9 than when it claims 0.5. Then selective prediction, which means answering only where you are confident and staying silent elsewhere, is impossible. The small model can no longer hide behind its confident slice, and the only remaining designs are a much larger model or no feature at all. Selective prediction is the idea the whole model-choice argument rests on.

The check is cheap and you should run it first: bucket held-out positions by predicted confidence, and plot observed accuracy against it. A useful calibrator gives you a line near the diagonal; a flat line kills the design.

State out loud: that perplexity is a release gate and never the headline, so that nobody spends a quarter on an 8% perplexity win that ships nothing.

Data and labels

With both objectives fixed, the next question is what they are trained on — and here the privacy machinery is part of the data architecture, not a compliance appendix.

Every sent message is a completed sequence, so the supervision is free and enormous — nobody has to label anything, because the user already wrote the right answer. Take a random cut point in each message: everything before it is the prefix, the next N tokens are the target.

sent message:  "Thanks for sending that over. I'll review it and get back to you Friday."
cut at token 8 ->  prefix:  "Thanks for sending that over. I'll review"
                   target:  " it and get back to you"

The pipeline, and what each stage is protecting against

Raw sent mail is not training data. Each stage below removes something specific, and skipping a stage leaves a specific failure behind. One term recurs in the table: an n-gram is simply a run of n consecutive tokens, so let me know if is a 4-gram; counting n-grams is how you find text that repeats across a corpus.

After all of it the corpus is roughly 300M messages with 4 cut points each, so 300M x 4 = 1.2B training examples.

How many tokens is that, and why the answer is two numbers

The token count needs one more constant, and it is the one that is easiest to leave out: the average surviving message is about 340 characters, so at 4 characters per token it is 340 / 4 = 85 tokens.

State that constant, because without it the token budget cannot be checked. With it, two different numbers both turn out to be defensible:

target-span tokens   1.2B examples x ~6.7-token target spans   =  8B tokens
tokens processed     300M messages x 85 tokens/message         =  25.5B tokens

The 8B figure counts only the spans being predicted. The 25.5B counts every token the model reads.

Which one belongs in a training-cost estimate depends on how the run is structured. Pack whole messages and compute loss on every position, and you get 25.5B. Sample four short targets per message and compute loss only there, and you get closer to 8B.

There is a tell worth knowing. If you ever see 8B quoted alongside 1.2B examples with no message length given, work out 8e9 / 1.2e9 = 6.7. That 6.7 is the target span length in tokens, not the tokens processed per example — which tells you which of the two conventions the author was using without them saying so.

The conclusion survives either way, which is the point of showing both. Priced below, the run is $33 of compute at 8B tokens and $106 at 25.5B — against a data pipeline, a canary suite and a calibration harness that cost engineer-quarters. The training run is rounding error at either number, so the ambiguity is worth naming and not worth resolving.

Assumptions in this stage.

State out loud: 4 cut points per message and 300M surviving messages, which together set the 8B-token training budget and therefore the training cost derived later.

Load-bearing: that sent mail is the right distribution to imitate. The feature assists composition, and every filter in the table above exists to enforce that — machine-generated mail and pasted documents are removed precisely because nobody typed them.

If the corpus were contaminated with a large fraction of text no human composed, the model would be fluent at exactly the sentences the feature will never be asked to complete. Worse, the failure would not show up in perplexity, because perplexity would be measured on the same contaminated held-out set.

What breaks: the diagnostic that catches this is per-position accuracy on real live prefixes, not held-out corpus perplexity. The two coming apart is the signal that the corpus has drifted from the traffic.

Privacy is not a compliance section, it is the data architecture

Four mechanisms, in increasing order of strength, make it possible to train on other people’s private mail — and each carries a measured quality cost, because a privacy mechanism quoted without its utility cost is not a design.

The corpus is other people’s private mail, so the architecture has to make certain failures impossible rather than unlikely.

1. Never log the prefix. The request payload is the user’s unsent email. A debug log of requests is a mail archive with a retention policy nobody signed off on. Log token counts, latencies, the trigger decision, and the outcome — never the text. This one is free and it is the one teams actually get wrong.

2. Federated learning. Federated learning means training a shared model without ever collecting the training data. Each participating device trains on the mail it already holds, and sends back only what it learned.

The thing it sends is an update, written dW below: the small change to the model’s weights that would make the model better at predicting that one user’s mail. It comes from the gradient, which is the direction of steepest improvement of the loss with respect to each weight.

Those updates are combined under secure aggregation — a cryptographic protocol in which each client’s contribution is masked, so the server can compute the sum over all clients but cannot read any individual term.

Net effect: the device’s data never leaves the device, and the server sees only a sum over at least 20,000 clients per round.

The diagram below traces one round of that loop. Follow it in four steps:

1. On the device, the local sent mail produces an update dW.
2. The update is clipped — scaled down if it is too large — by the rule dW <- dW · min(1, S/norm(dW)). Read that rule as: if the update is smaller than the ceiling S, leave it alone; if it is bigger, shrink it to exactly S.
3. The clipped updates go into secure aggregation, and Gaussian noise is added to the resulting sum.
4. The noisy average is applied to the global model, which is then sent back to the devices for the next round.

Only step 1 touches user text, and step 1 happens on hardware the user owns.

flowchart TD
    subgraph DEV["On device — data never leaves"]
        D1["Local sent mail"] --> D2["Compute update dW<br/>on the local corpus"]
        D2 --> D3["Clip: dW <- dW · min(1, S/norm(dW))"]
    end
    D3 --> AGG["Secure aggregation<br/>server sees only the SUM<br/>over m >= 20,000 clients"]
    AGG --> NZ["Add Gaussian noise<br/>N(0, sigma^2 · S^2)"]
    NZ --> UP["Average and apply<br/>to the global model"]
    UP -.->|next round| D2

    style DEV fill:#2d6a4f,color:#fff
    style NZ fill:#bc6c25,color:#fff
    style AGG fill:#1d3557,color:#fff

3. Differential privacy, and the mechanism that actually does the work. Differential privacy (DP) is a formal guarantee that the trained model would have come out very nearly the same had any one user’s data been absent — so nothing specific to that user can be recovered from it.

It is delivered by two steps, in this order:

1. Clip each client’s update to L2 norm S. The L2 norm is the ordinary geometric length of the update viewed as a vector: the square root of the sum of its squared components. Clipping means no client can push the model further than S, no matter how unusual their mail.
2. Add Gaussian noise to the aggregate. That means adding random values drawn from a bell curve centred on zero, whose spread is set in proportion to S.

The clip bounds any single user’s influence. The noise makes that bound probabilistic rather than merely small.

Neither step works without the other. Noise without clipping is defeated by one outlier client whose enormous update swamps it. Clipping without noise still leaks, because a bounded contribution is still a contribution you can look for.

Why differential privacy needs a large population

The cost of DP is measurable, and the shape of that cost explains why the technique only works at scale. In the arithmetic below, sigma is the noise scale, S the clipping ceiling, and m the number of clients whose updates are summed in one round:

noise std added to the sum      =  sigma · S           (independent of m)
signal in the sum               ~  m · (mean update)
relative noise                  ~  sigma · S / (m · mean)      ->  falls as 1/m

Read those three lines as one argument. The noise added is the same size regardless of how many clients contributed. The signal grows in proportion to the number of clients. So the ratio between them shrinks as clients are added.

The noise penalty is inversely proportional to the number of clients per round, so differential privacy is affordable only if you have millions of users — which is exactly the situation where you also most need it.

What an epsilon actually means, and the five numbers behind it

At m = 20,000 clients per round and sigma = 1.0, over T = 3,000 rounds, you land near (epsilon = 8, delta = 1e-9). Those two symbols are the standard way a DP guarantee is quoted:

• epsilon caps how much more likely any observable outcome becomes because one particular user was included. Smaller is stronger; 8 is a moderate real-world setting rather than a strict one.
• delta is the small probability that the epsilon bound fails to hold at all — here one in a billion.

A quoted epsilon is meaningless without the two things that produced it, so state them.

First, the accountant — the method used to add up privacy loss across rounds. Composing 3,000 rounds naively multiplies the per-round loss and gives an epsilon in the hundreds, which is why nobody does that. The number above comes from a Rényi differential privacy accountant (equivalently, the moments accountant), which tracks the loss as a Rényi divergence across orders and converts once to (epsilon, delta) at the end. That is what makes growth in T behave like sqrt(T) rather than T.

Second, the subsampling rate q — the fraction of the population touched per round. Here 20,000 clients drawn from 50M users gives q = 20,000 / 50,000,000 = 4e-4. (This q is standard notation in the DP literature and has nothing to do with the q used from the trigger policy onward, which is a calibrated confidence. The collision is unfortunate and universal.)

Subsampling is not a detail. Privacy amplification by subsampling is most of where the guarantee comes from, because a user who is not sampled in a round incurs no privacy loss in that round, and the per-round loss scales roughly with q. Raise q to 4e-3 by sampling 200,000 clients per round and the guarantee degrades sharply, even though every other symbol on the page is unchanged.

So the honest form of this claim is five numbers, not two: (epsilon = 8, delta = 1e-9) under RDP accounting, at q = 4e-4, over 3,000 rounds, with sigma = 1.0.

One sanity check is worth doing every time. Delta should sit well below 1/N, where N is the population, because a delta near 1/N permits the mechanism to leak one whole user. Here 1/N = 1/50,000,000 = 2e-8, and 1e-9 clears it by a factor of 20.

Measured cost on our evaluation set: +3.1% perplexity, which propagates to -1.4 points of acceptance rate. Say that number out loud. “We would add differential privacy” with no utility cost attached is not a design, it is a slogan.

4. What differential privacy is actually buying, and how to measure it. The thing being bought is not a compliance checkbox, it is protection against memorization — a model reproducing a specific rare string from its training data verbatim rather than generalising from it. A language model trained on raw mail will, given the prefix my account code is , complete it with a real account code belonging to a real user.

The measurement is canary insertion. A canary is a synthetic secret — a made-up string of the same shape as the real thing — planted in the training corpus a controlled number of times before training starts.

After training, you measure how strongly the model prefers the true canary over randomly generated alternatives of the same shape, and report that preference in bits of exposure. One bit means the model halves the search space for an attacker. So 14 bits means it has narrowed roughly 2^14 = 16,384 candidates down to one, and 23 bits means roughly 8.4 million down to one. “Below floor” means the preference is indistinguishable from chance.

The table below plants the canary at three frequencies and reads exposure with and without differential privacy.

Read the middle row as the important one: a secret that appears just 8 times in 300M messages is recoverable from an undefended model, and differential privacy pushes it back below the detection floor. A canary suite is the only thing that turns “we take privacy seriously” into a number, and it runs on every training run like any other test.

5. The output-side control: k-anonymity on suggestion strings. k-anonymity here means never emitting a string unless at least k different people independently wrote it, so no output can be traced back to one person. The setting used is k = 50.

Mechanically: independently of the model, maintain a whitelist of n-grams that appear in the sent mail of at least 50 distinct users, and refuse to emit any suggestion that is not on it.

That check is a filter in code, not an instruction to the model, and chapter 07 has exact vocabulary for the difference (Advisory vs enforcement mechanically):

• Enforcement is a branch in the harness. Its probability of violation is exactly zero.
• Advisory is a rule written into the token stream. It shifts the model’s logits toward compliance, but can never drive them to zero.

The whitelist is enforcement. Use the word verbatim in an interview, because the distinction is mechanical rather than a matter of emphasis: it makes the memorization failure unrepresentable rather than unlikely.

Measured cost: coverage falls about 9%, because genuinely correct but idiosyncratic completions get blocked too. That is the right trade.

Assumptions in this stage.

Load-bearing: that there are millions of users. Every mechanism here scales with population. Differential privacy’s noise penalty falls as 1/m in the number of clients per round, and k-anonymity needs 50 distinct users to have written a phrase before it may ever be suggested.

What breaks: at 200,000 users rather than 50 million, the DP noise at a comparable epsilon would swamp the signal, and the k-anonymity whitelist would be so sparse that coverage collapses far below the 9% loss quoted above.

The design for a small population is a different design: no federated training, no whitelist, an aggressively filtered public corpus, and a much narrower suppression policy.

State out loud: that the +3.1% perplexity cost of differential privacy is measured on your evaluation set and not borrowed from a paper.

Model choice, derived from the latency budget

The model size is settled by arithmetic on the latency budget, before quality enters the conversation at all.

How much of the 100 ms is left for the model

Work backwards from the 100 ms. Subtract the costs that are fixed no matter which model you pick — the debounce wait before a request is even issued, the network hop if the model is on a server, and the time to paint pixels on screen. Every one of those numbers is itemised later in the deployment comparison; pulled forward here, the subtraction is:

budget                                        100 ms
- debounce (wait for typing to pause)          40 ms
- network round trip (18 out + 16 back)        34 ms
- render                                        2 ms
                                             --------
  left for the model                          ~24 ms

So call it ~25 ms for decode. That is the strictest form of the budget: it assumes the model is on a server, and a server hop is the case any candidate has to survive if it cannot fit on a phone. On the device the 34 ms of network disappears and roughly 58 ms is available instead — which is why the chosen design can spend 26.5 ms on decode and still land at a 78 ms p99. Either way, the numbers below either fit in tens of milliseconds or they do not fit at all.

Why decode time is a division problem

Decode is not limited by arithmetic. It is limited by memory bandwidth: the number of bytes per second the accelerator can pull out of its own memory. Each decode step has to read every weight it uses and does very little maths with each one, so the accelerator spends its time waiting on memory, not computing (2b latency is dominated by decode and decode is sequential).

That makes the time for one decode step just bytes read divided by bytes per second:

step_time  =  (weight_bytes + batch × context × kv_per_token) / memory_bandwidth

The numerator is the two things that must be read on every step:

• weight_bytes — all of the model’s weights.
• batch × context × kv_per_token — the KV cache, meaning the stored key and value vectors for every token already in the context. Attention re-reads them at every step so that it does not have to recompute them. batch is how many users’ requests are served together in one pass, which is worth doing precisely because they share the cost of reading the weights.

The four candidates, priced

Fix three settings: batch 64, 700-token contexts, and 3.3 TB/s of memory bandwidth — an H100-class accelerator, the same reference part used throughout this folder. Two more terms appear in the table:

• GQA is grouped-query attention, an arrangement in which several query heads share one key/value head. “GQA 4” means 4 key/value heads. Fewer key/value heads means a proportionally smaller KV cache to re-read.
• int8 and fp16 are the numeric formats the weights are stored in, at 1 byte and 2 bytes per weight respectively.

Each row of the table is one division, and the last two columns are the verdict.

Nothing in that table is quoted from a benchmark. Every cell is one of two substitutions, so work both of them on one row.

Take the 70B row, since it is the one that decides the argument. First the kv/token column, which is 2 (a key and a value) x layers x kv_heads x head_dim x bytes_per_number:

kv/token   =  2 x 80 layers x 8 kv heads x 128 head_dim x 2 bytes
           =  327,680 bytes  =  320 KB per token of context

Then the step time, substituting into the formula above:

weight_bytes  =  70e9 params x 2 bytes            =  140,000,000,000 B
KV bytes      =  64 batch x 700 ctx x 327,680 B   =  14,680,000,000 B
                                                     ---------------
                                 total to read       154,680,000,000 B

step_time  =  1.5468e11 B / 3.3e12 B/s  =  0.0469 s  =  46.9 ms
6 tokens   =  6 x 46.9                              =  281 ms

The other two kv/token figures come out of the same multiplication with the standard shapes for their sizes — 24 layers at head_dim 64 for the 300M specified below, and 32 layers at head_dim 128 for the 7B: 2 x 32 x 8 x 128 x 2 = 131,072 B. Without those layer counts, the 131 KB and 320 KB would be numbers you had to take on trust, and the KV term is half of the step-time formula.

A 70B model spends the whole 100 ms budget 2.8 times over on decode alone, before a single packet moves (281 / 100 = 2.8). The 7B’s 36 ms already exceeds the ~25 ms of decode the budget allows. The 300M int8 comes in at 2.5 ms, about 10x under.

That is the model selection argument, and it is arithmetic rather than preference. Note what does not appear anywhere in it: quality. The large models are excluded before quality is discussed at all.

The model that fits

One model survives that budget, and its parameter count is worth adding up so you can see where the 300M goes.

The model is a decoder-only transformer. “Decoder-only” means it does exactly one thing — predict the next token from the tokens before it — with no separate module for reading an input.

Its configuration, term by term:

Each layer holds two blocks:

• Attention has four weight matrices — query, key, value and output, written Q, K, V and O. The K and V matrices are only a quarter the width of Q and O, because there are 4 key/value heads instead of 16, and 1024 x (4/16) = 256.
• The FFN, or feed-forward network, is the two-matrix block that follows attention in every transformer layer and does the per-token processing between attention steps. It expands 1024 dimensions to 4096 and back, which is two matrices of 1024 × 4096.

The block below adds those up to the headline 300M, and where the parameters actually sit is worth noting: the FFN is roughly three quarters of every layer.

per layer:  attn  1024x1024 (Q) + 1024x256 (K) + 1024x256 (V) + 1024x1024 (O)  = 2.62M
            ffn   2 x 1024x4096                                                 = 8.39M
                                                                                 -------
                                                                                 11.0M
total    =  24 x 11.0M  +  32,000 x 1024 (tied embedding)  =  297M parameters
kv/token =  2 x 24 x 4 x 64 x 2 bytes                      =  24 KB

Checking the two lines that matter: 24 x 11.0M = 264M, plus 32,000 x 1024 = 32.8M of embedding, gives 297M — which is what “300M” is rounding.

The kv/token line counts the bytes the KV cache holds per token of context: 2 (one key and one value), times 24 layers, times 4 key/value heads, times 64 numbers per head, times 2 bytes per number, giving 24,576 bytes. That 24 KB is the figure the step-time table used, and it reappears in the server memory calculation later.

This model is trained from scratch on the mail corpus rather than fine-tuned from a general-purpose model, because the distribution is narrow, the vocabulary is domain-specific, and at 300M parameters capacity spent on general world knowledge is capacity not spent on this task.

Why decoder-only and not an encoder-decoder — the load-bearing argument

The single most important architectural decision in the chapter comes down to one property of causal attention.

The alternative framing is an encoder-decoder: one module (the encoder) reads the input and builds a representation of it, and a second module (the decoder) writes the output while consulting that representation. It is the obvious shape for “prefix in, continuation out”, and it is wrong here for a mechanical reason.

In an encoder-decoder, the encoder is bidirectional over the source, meaning each token is allowed to look both backwards and forwards. Token i’s representation therefore depends on tokens that come after i. Append one character to the composed prefix and every source representation changes, so the entire encoder must re-run from scratch.

In a causal decoder, attention is masked so that each token may only look at itself and the tokens before it. Token i’s key and value vectors therefore depend on tokens 0..i and nothing after (Prompt caching derived). Appending a token leaves every previous key and value bit-for-bit identical, which is precisely what makes them cacheable.

The user extends the prefix on every keystroke. So this is not a stylistic preference — it is the difference between re-encoding 700 tokens per keystroke and encoding 4.

The two diagram branches below are the same keystroke, t+1, under each architecture. In the causal decoder, tokens 0..n have their keys and values already cached and unchanged, so only the four new tokens n+1..n+4 need to be run through the model — a phase called prefill, which processes a run of known tokens in one parallel pass — and then 6 tokens are decoded. In the bidirectional encoder, every representation depends on every other, so all 700 tokens must be re-encoded with nothing reusable before the same 6 tokens are decoded.

flowchart TD
    subgraph DEC["Causal decoder — keystroke t+1"]
        A1["tokens 0..n<br/>K,V already cached<br/>UNCHANGED"] --> A2["new tokens n+1..n+4<br/>prefill only these"]
        A2 --> A3["decode 6 tokens"]
    end
    subgraph ENC["Bidirectional encoder — keystroke t+1"]
        B1["tokens 0..n+4<br/>every representation<br/>depends on every other"] --> B2["re-encode ALL 700<br/>nothing reusable"]
        B2 --> B3["decode 6 tokens"]
    end

    style A1 fill:#2d6a4f,color:#fff
    style A2 fill:#40916c,color:#fff
    style B2 fill:#9d0208,color:#fff

The fills encode three tiers of work, and the third one is the whole argument:

• Dark green is zero work — tokens 0..n, whose keys and values are already cached and are bit-for-bit unchanged by the append.
• Light green is small work — the four new tokens, the only thing that has to be prefilled.
• Red is all the work — all 700 source representations, recomputed because a bidirectional encoder gives every one of them a dependency on every other.

Read the two branches as a pair and the comparison is not “green versus red”. It is “zero plus small, versus all” — which is the 4-against-700 ratio priced immediately below.

decode 6 tokens appears in both branches and is identical in both: whatever you did to build the representations, writing the suggestion costs the same 6 decode steps either way. It is the constant both sides share, which is what makes the difference between them attributable entirely to the encoding strategy.

At fleet scale that 4-against-700 ratio is the whole compute budget. Two things to know before reading the arithmetic:

• FLOP stands for floating-point operation, the unit of arithmetic work. FLOP/s is the rate.
• Prefilling one token through a model costs about 2 × params operations — one multiply and one add per parameter. That rule of thumb is worth memorising; it is the basis of nearly every capacity estimate in this folder.

peak load                          500,000 requests/s   (derived under Scale, below)
prefill FLOPs = 2 x params x tokens

re-encode 700 tokens each:   500k x 700 x 2 x 3e8  =  2.1e17 FLOP/s  ->  700 accelerators
prefill 4 new tokens each:   500k x   4 x 2 x 3e8  =  1.2e15 FLOP/s  ->    4 accelerators
                                                                          ----------------
                                                                          175x

Working the first line by hand: 500,000 x 700 = 3.5e8 tokens prefilled per second, and each token costs 2 x 3e8 = 6e8 FLOP, so 3.5e8 x 6e8 = 2.1e17 FLOP/s. Divide by 300 TFLOP/s (3e14) per accelerator and you get 700 of them. The second line is the same arithmetic with 4 tokens instead of 700, so it lands at 4 accelerators.

The reference part throughout is the same H100-class accelerator as 2b latency is dominated by decode and decode is sequential: 80 GB of memory, 3.3 TB/s of bandwidth, 300 TFLOP/s effective, $2.50 per GPU-hour. TFLOP/s is trillions of floating-point operations per second.

The 175x itself does not depend on any of that, though. It is just 700 / 4, and it survives whatever hardware you plug in.

Assumptions in this stage.

State out loud: batch 64, 700-token contexts, and an H100-class part at 3.3 TB/s. Change the hardware and every step time moves, but the ordering of the four candidates does not, because that ordering is set by weight bytes.

Load-bearing: that the decode budget really is about 25 ms — the 100 ms minus the fixed costs you cannot compress. It is worth checking that the conclusion is not fragile to that number. Remove the debounce entirely and the budget grows by 40 ms, and the 7B’s 36 ms is still excluded once you add the network back. Make the network hop free and the server design still fails on its p99.

Also load-bearing: that the context is around 700 tokens. At 4,000 tokens of thread context, the KV term stops being small next to the weight term — for the 300M, 64 x 4000 x 24,576 = 6.3 GB of KV against 0.3 GB of weights. That is exactly why the long-thread case is served by a separate tier on a relaxed 400 ms budget rather than by the same path.

The quality tradeoff, and why it is smaller than it looks

The honest objection is that a 300M-parameter model is much worse than a 7B one. That is true in aggregate and mostly irrelevant here, because aggregate quality is the wrong measurement for a system that gets to stay silent — a claim worth deriving rather than asserting.

Measure top-1 agreement: for each position, take the single highest-probability next token from the 300M model and from a 7B model, and ask how often they are the same token. The 7B here is acting as a teacher — a larger model whose behaviour is treated as the reference the smaller student model is compared against.

Measure it twice: once over all positions, and once over only the positions where the small model is confident. The gap between the two rows is the entire argument.

The trigger only ever fires in that second regime. The calibrated rule derived below is not literally “raw top-1 probability above 0.85”, but it selects a slice of the same size — 12-14% coverage — and of the same character.

On that slice the suggestion is drawn from the head of the distribution, where a small model and a large one agree. So the 29 points of aggregate disagreement (100 - 71) collapse to 4 points (100 - 96) on the suggestions actually shipped. The 29 is the number an interviewer will quote at you; the 4 is the number the user experiences.

Selective prediction — the freedom to answer only when confident and abstain otherwise — is what makes a small model defensible. The model is allowed to be bad, as long as it knows when.

Distillation recovers part of the remaining gap for free. Instead of training the 300M student only on which token actually came next, train it to match the 7B teacher’s full probability distribution over its top-k candidates at each position — the teacher’s logits, meaning the raw scores it produces before they are normalised into probabilities. A single sampled token tells the student one fact per example; a distribution over the top candidates tells it what the teacher thought of every plausible alternative, which is far more signal for the same corpus.

Training

The training run has a price, and the point of the number is how small it is.

Training cost follows a standard rule of thumb: a full training step — forward pass, backward pass and weight update — costs about 6 × params × tokens floating-point operations. That is roughly three times the 2 × params × tokens of a forward pass alone, because the backward pass costs about twice what the forward pass did.

FLOPs  ≈  6 x params x tokens  =  6 x 3e8 x 8e9  =  1.44e19
8 accelerators at 300 TFLOP/s effective  =  2.4e15 FLOP/s   ->  1.67 hours
                                                            ->  13.3 GPU-hours

Step by step: 8 accelerators at 3e14 FLOP/s each is 2.4e15 FLOP/s combined. Then 1.44e19 / 2.4e15 = 6,000 seconds, which is 1.67 hours. Multiply by 8 accelerators to get 13.3 GPU-hours.

That is about 13.3 x $2.50 = $33 of compute.

On the other reading of the token budget — 25.5B tokens processed rather than 8B target-span tokens, the two figures reconciled above — the same arithmetic gives 42.5 GPU-hours and $106. Quote whichever you can defend; the conclusion is identical, which is exactly why the ambiguity is safe to leave open.

Say the conclusion out loud, because it reframes the project: the model is not the expensive part. The data pipeline, the canary suite, the counterfactual replay harness and the trigger calibration are the work.

Three things that go on top of the base run

1. Int8 post-training quantization is the highest-return change in the whole project.

Quantization means storing each weight in a smaller numeric format — here 8-bit integers instead of 16-bit floats. Post-training means doing it after training rather than training in that format, so it takes hours rather than a full rerun.

What it buys:

The device row is the one to look at. Halving the bytes halves the memory traffic, and decode is bound by memory traffic, so the device time halves almost exactly. The server row halves by less because the KV cache — which did not shrink — is a large share of the bytes read at batch 64.

2. Per-language models are trained separately, for the capacity reason given in the data pipeline table: at 300M parameters, capacity is the binding constraint, so one model per language beats one multilingual model of the same size.

3. On-device personalization is a small adapter — a thin set of extra weights trained on top of the frozen base model, cheap enough to fit on a phone. It is trained on the individual’s own sent mail, on their device, and never uploaded.

Personalization is where acceptance rate actually moves, because people have idiosyncratic sign-offs and phrasings. It is also the version with no privacy exposure at all, because nothing leaves the device.

Assumptions in this stage.

State out loud: 8 accelerators for 1.67 hours, which is the whole training bill.

Load-bearing: that 8B tokens is enough to train 300M parameters to convergence on this narrow distribution. That ratio is 8e9 / 3e8 ≈ 27 tokens per parameter — comfortable for a domain-specific model, thin for a general one. The reason it holds here is that the distribution is narrow, and the reason it would fail is the same one that argues against a multilingual model.

What breaks if training is expensive instead of trivial: nothing about the architecture, but the project’s centre of gravity moves. If the model cost $3M rather than $33, retraining on a fresh corpus each quarter stops being free, and the drift guard in failure mode 7 becomes a budget item rather than a habit.

The trigger policy — a precision problem, derived

Now the heart of the design: the language model decides what to say, and the trigger policy decides whether to say it.

The model can always produce a continuation. The decision is whether to show it, and how long to make it — and because the two error types cost wildly different amounts, that decision has an exact answer rather than a tuned one.

The cost asymmetry, priced

Everything that can happen once a suggestion is shown fits in one table, and the threshold falls straight out of pricing it.

The four outcomes are measured in different units — seconds of typing saved, seconds of attention spent, the risk of losing a user entirely. To compare them, normalise everything into one made-up unit called a utile.

Define it by anchoring it: 1.00 utile is the value of one average accepted suggestion, which is 18 characters saved, or about 5 seconds of typing. Every other number in this section is priced against that anchor.

The letter q throughout means the calibrated probability that the suggestion is correct.

Here are the four things that can happen once a suggestion is shown. Read the last row first — it is 30x the size of anything else in the table, and it is what the whole design is built around.

Five symbols come out of that table and are used for the rest of the chapter:

Solving for the threshold

Show the suggestion when its expected value is positive — that is, when the expected gain from the times it is right exceeds the expected loss from the times it is not:

q · A_ok · V_a   >   (1 - q) · (C_r + A_wrong · C_w)

Substitute the five values above, then solve for q:

q · 0.55         >   (1 - q) · (0.05 + 0.03 x 30)     substitute
q · 0.55         >   (1 - q) · 0.95                   0.03 x 30 = 0.90, + 0.05 = 0.95
0.55 q           >   0.95 - 0.95 q                    expand the right side
0.55 q + 0.95 q  >   0.95                             collect the q terms
1.50 q           >   0.95

q*  =  0.95 / 1.50  =  0.633

So the rule is: show the suggestion when the calibrated probability of being right is at least 0.633.

That result is exactly the standard cost-matrix threshold C_fp / (C_fp + C_fn) (Choosing a threshold from the cost matrix):

• C_fp is the cost of a false positive — acting when you should not have, here showing a suggestion that turns out wrong. It is C_r + A_wrong · C_w = 0.05 + 0.90 = 0.95.
• C_fn is the cost of a false negative — failing to act when you should have, here staying silent on a suggestion that would have been accepted. It is A_ok · V_a = 0.55, the value forgone.

And 0.95 / (0.95 + 0.55) = 0.633, the same number by the same route.

Compare it against the two thresholds you might have guessed

The difference between these three rules is the entire argument of this section.

The gap between 0.083 and 0.633 — a factor of 7.6 — is entirely the A_wrong · C_w term, a 3% chance of a bad accept priced at 30 times a good one. That one term is 0.90 of the 0.95 that makes up C_fp.

A candidate who reasons only about typing time lands almost an order of magnitude low. Every extra suggestion that low threshold buys comes from the low-q region, which is exactly where the bad-accept term dominates.

And the failure shows up as feature-disable rate rather than as anything on the quality dashboard, which is why it can run for a quarter unnoticed.

Assumptions in this stage. This is where user behaviour, rather than traffic or hardware, becomes load-bearing.

Load-bearing: A_wrong = 0.03, the rate at which people accept a suggestion that is wrong. It is the term that moves the threshold from 0.083 to 0.633, so the design is more sensitive to this one behavioural number than to any hardware number in the chapter. Two counterfactuals:

A_wrong = 0.005 (users who always read):  C_fp = 0.05 + 0.005 x 30 = 0.20
                                          q*   = 0.20 / (0.20 + 0.55) = 0.27
                                          coverage roughly doubles

A_wrong = 0.10  (heavy autopilot):        C_fp = 0.05 + 0.10 x 30  = 3.05
                                          q*   = 3.05 / (3.05 + 0.55) = 0.85
                                          coverage collapses

Also load-bearing: C_w = 30, the price of a bad accept. That is a judgement about how much a user-visible mistake costs in eventual churn, not a measurement.

What you do about it: both are measurable online. A_wrong comes from the retraction rate described under Guardrails; C_w comes from the feature-disable rate that follows a bad accept. So the first launch is deliberately conservative and exists mainly to measure them.

q is not the model’s sequence probability

The number fed to that threshold cannot be the model’s own confidence, and something small has to be built instead.

The obvious candidate for q is the sequence probability: multiply together the model’s probability for each token of the suggestion. It fails twice.

First, it is uncalibrated. The raw number a model reports does not behave like a real probability, so among the suggestions it scores 0.7, the fraction that turn out correct is not 70% (Calibration what it means and when it matters).

Second, and worse, it is length-confounded. It is a product of numbers below 1, so it shrinks with every additional token regardless of quality. A fixed threshold on it therefore silently caps how long a suggestion can be.

Watch that happen. Take a model whose mean per-token probability is 0.92 — a good model — and raise it to successive powers:

mean per-token probability 0.92:
    3 tokens -> 0.78    5 tokens -> 0.66    6 tokens -> 0.61    10 tokens -> 0.43

A threshold of 0.633 applied to the raw sequence probability makes 5 tokens the longest suggestion that can ever fire. 0.92^5 = 0.659 clears the bar and 0.92^6 = 0.606 does not — for reasons that have nothing to do with whether long suggestions are good.

That is not a hypothetical problem. The optimum length derived two subsections below is 5-6 tokens on the fired slice, so the uncalibrated threshold is truncating the answer exactly at the boundary where it matters.

The calibrator

The fix is a second, tiny model called a calibrator, trained on held-out data to predict the one thing the threshold actually needs: P(the user's actual continuation starts with this suggestion).

It has to be small enough to be free at inference, because the latency budget is already spent. A logistic regression works, as does a shallow GBDT — gradient-boosted decision tree, an ensemble of small trees where each tree corrects the previous one’s errors.

Its inputs are features derived from the language model’s output, not from the text itself. The table below lists them; the top three are the ones that carry most of the signal.

Two of those features need unpacking.

The margin is the gap between the best and second-best candidate continuations under beam search — a decoding method that keeps several partial continuations alive at once, instead of committing to the single best token at each step. A large margin means the model saw one obvious continuation rather than three near-ties.

Entropy is the standard measure of how spread out a probability distribution is: near zero when one token has almost all the probability, large when many tokens are plausible. Entropy at the first token — the branching point — turns out to predict failure better than the probability of the path that was eventually chosen, because that is where the model actually commits.

Then threshold the calibrated q at 0.633. Coverage — the fraction of evaluated positions at which anything is shown — lands near 12-14%.

Length is the same decision, not a separate one

“How long should the suggestion be” is not a second question — it is the same expected-utility calculation with the length left free. Answer it on the wrong population and the answer comes out wrong by a factor of five.

Choose the length L that maximises expected utility U(L). Two forces pull against each other:

• The value of an accept grows with length, because more characters are saved. At 4.2 characters per token, an L-token suggestion saves 4.2L characters, which is 4.2L / 18 utiles.
• The probability of being right shrinks with length, because every additional token is another chance to be wrong.

q(L) is that shrinking correctness: the probability that all L tokens are right, which is the running product of the per-position accuracies. So q(3) = acc_1 × acc_2 × acc_3.

Put both forces in one expression. The 0.95 is C_fp from the threshold derivation above — the cost of a suggestion that is shown and turns out wrong:

U(L)  =  q(L) · A_ok · (4.2L / 18)  -  (1 - q(L)) · 0.95
         \_______ gain _________/      \____ cost ____/

Run that calculation twice, on two different populations, because the difference between them is the whole lesson.

Run 1: over all contexts. The measured per-position accuracies are 0.94, 0.91, 0.88, 0.84, 0.79, 0.72 — the probability that the 1st, 2nd, 3rd and later tokens of a suggestion are each individually correct.

Work the L = 2 row by hand so the rest of the table reads itself:

q(2)   =  0.94 x 0.91                  =  0.855
chars  =  4.2 x 2                      =  8.4
gain   =  0.855 x 0.55 x (8.4 / 18)    =  0.220
cost   =  (1 - 0.855) x 0.95           =  0.137
U(2)   =  0.220 - 0.137                =  +0.082

That table says never suggest more than two tokens, and treat anything past three as net harmful.

That conclusion is wrong, and understanding why is the point of this subsection.

It is the unconditional optimum — the best length averaged over every position in every message. But the trigger never fires on an average position. It fires on the 12% of positions where the model is confident, and those positions have much higher per-position accuracies.

Run 2: over the fired slice only. The measured per-position accuracies there are 0.99, 0.98, 0.97, 0.96, 0.94, 0.92, 0.90, 0.87, 0.83, 0.78. Same formula, same A_ok, same C_fp — only the accuracy inputs changed:

The optimum moves from 2 tokens to 5-6, and peak utility rises from +0.082 to +0.402 — a factor of 0.402 / 0.082 = 4.9. That entire difference is the value of the trigger.

It is also the cleanest statement of what selective prediction buys: not a better model, but permission to be more ambitious in the places where you are already right.

The two functions below are the whole policy in code. should_suggest prices one candidate at a given length and says whether to show it; best_length sweeps L upward, accumulating q(L) as the running product of per-position accuracies, and returns the length with the highest utility. Note that the threshold is computed inside the function rather than passed in — the point of the derivation is that it is not a tunable.

def should_suggest(q: float, length_tokens: int, *, a_ok=0.55, a_wrong=0.03,
                   c_read=0.05, c_wrong=30.0, chars_per_token=4.2,
                   chars_per_utile=18.0) -> dict:
    """Expected-utility trigger. q must be CALIBRATED, not a raw sequence prob.

    Reduces to the cost-matrix threshold C_fp / (C_fp + C_fn) when the value of
    an accept is held at 1.0; the length term is what makes it a joint decision.
    """
    c_fn = a_ok                                   # value forgone by not showing
    c_fp = c_read + a_wrong * c_wrong             # cost of showing a wrong one
    threshold = c_fp / (c_fp + c_fn)

    value = chars_per_token * length_tokens / chars_per_utile
    utility = q * a_ok * value - (1 - q) * c_fp
    return {"show": q >= threshold and utility > 0,
            "threshold": threshold, "utility": utility}


def best_length(per_position_acc, **kw):
    """Pick L jointly with the show/no-show decision. Returns (L, utility)."""
    best, q = (0, 0.0), 1.0
    for length, acc in enumerate(per_position_acc, start=1):
        q *= acc
        u = should_suggest(q, length, **kw)["utility"]
        if u > best[1]:
            best = (length, u)
    return best

Assumptions in this stage.

Load-bearing: that the per-position accuracies on the fired slice really are the second set of numbers and not the first. The optimal length is 5-6 tokens only because the trigger has already selected a confident population. If calibration were poor, so that the fired slice looked like the average slice, the correct length would be 2 tokens and the feature would save roughly a third as much.

State out loud: 4.2 characters per token and 18 characters per accepted suggestion, which together convert a token count into a utile.

What breaks: both length tables are computed on a single population, and both must be re-derived per language. A tokenizer that fragments a script produces a completely different characters-per-token figure — the failure traced in Non-Latin scripts below.

Hard suppression, before any of the above

A few classes of suggestion never reach the trigger at all — and even the ones that look like policy decisions fall out of the same arithmetic.

Some classes are never worth their expected value at any confidence the system can certify, so they are excluded in code before the trigger is consulted at all. Every row below is enforcement rather than advisory in chapter 07’s sense — a branch in code, not a sentence in a prompt — so it cannot be talked out of it:

One phrase in that table needs a plain definition. A protected attribute is a characteristic that law and policy forbid making decisions on — race, religion, disability, sexual orientation, national origin and the like — and the proximity rule suppresses suggestions near any such term because the corpus contains real-world associations that a completion would faithfully reproduce.

The gendered-pronoun rule falls straight out of the same expected-utility formula, which is why it belongs here and not in a values statement.

Two inputs change from the general case:

• The value shrinks. A pronoun saves about 3 characters, so V_a = 3/18 = 0.17 utiles instead of 1.00.
• The cost of a bad accept grows. A pronoun error is a personal misattribution about a named colleague, in a message the user then sends. Price it at C_w = 400 instead of 30.

Hold A_wrong = 0.03 and C_r = 0.05, and run the same cost-matrix ratio C_fp / (C_fp + C_fn):

C_fp  =  0.05 + 0.03 x 400        =  12.05
C_fn  =  A_ok x V_a  =  0.55 x 0.17  =  0.094

q*  =  12.05 / (12.05 + 0.094)
    =  0.992

Compare that to the general threshold of 0.633. A tiny payoff divided by a large risk pushes the required confidence to 99.2%.

A calibrated 99.2% on a demographic inference the model is making from names and job titles is not attainable, and it would not be trustworthy even if the number claimed it was. When the derived threshold exceeds what calibration can certify, the correct policy is to suppress the class entirely — which is the real-world design decision, recovered from arithmetic rather than adopted from a headline.

Serving

With the policy fixed, follow one keystroke through the running system. Two facts will decide where the model lives: the KV cache is a throughput lever rather than a latency lever, and holding those caches on a server costs more hardware than the computation itself.

The diagram traces the path, and it is easiest to read as four gates in a row: two that drop work before the model runs, the model, and two that drop work after.

Before the model — two cheap refusals. Each keystroke first increments a sequence number, a counter used later to tell whether a result that arrives is still about the text currently on screen. The keystroke then meets the debounce gate — 40 ms of quiet, and at a word boundary — and a keystroke that fails it is simply dropped, with no request issued at all. Survivors go through the hard suppression rules, and a blocked one is dropped there. Both of these cost almost nothing and are placed first for exactly that reason.

The model, and the number it produces. What is left reaches the on-device model, a 300M int8 network whose KV cache is already warm for this composition. Its output goes to the calibrator, which turns raw model scores into q. Note that the model has to decode all 6 tokens before either of the remaining checks can run — nothing downstream can start early — which is why the decode budget was the first thing fixed in this chapter.

After the model — two more refusals, and only then pixels. The trigger asks whether q clears 0.633 and whether the utility of the chosen length is positive; if not, the system shows nothing. If it passes, one final check asks whether the sequence number is still current — if it is not, the result describes text the user has already typed past, so discard it. Only then does the client render the grey inline text.

The dotted path is the one exception to all of the above. For the first suggestion in a long reply thread only, a server tier holding the full thread context on a relaxed 400 ms budget produces the candidate instead, and it rejoins the path at the same calibrator — so everything after the calibrator is identical whichever tier produced the candidate.

flowchart TD
    K([Keystroke]) --> SEQ["Increment sequence no."]
    SEQ --> DB{"Debounce:<br/>40 ms quiet AND<br/>at a word boundary?"}
    DB -->|no| DROP1["Drop. No request."]
    DB -->|yes| SUP{"Hard suppression<br/>rules"}
    SUP -->|blocked| DROP2["Drop"]
    SUP -->|clear| ODM["On-device model<br/>300M int8<br/>KV cache warm for<br/>this composition"]
    ODM --> CAL["Calibrator -> q"]
    CAL --> TRG{"q >= 0.633<br/>and U L > 0?"}
    TRG -->|no| DROP3["Show nothing"]
    TRG -->|yes| FRESH{"seq no. still<br/>current?"}
    FRESH -->|no| DROP4["Stale. Discard."]
    FRESH -->|yes| SHOW([Render grey inline])
    ODM -.->|first suggestion in a<br/>long thread only| SRV["Server tier<br/>full thread context<br/>relaxed 400 ms budget"]
    SRV -.-> CAL

    style DB fill:#bc6c25,color:#fff
    style SUP fill:#9d0208,color:#fff
    style ODM fill:#2d6a4f,color:#fff
    style TRG fill:#1d3557,color:#fff
    style FRESH fill:#bc6c25,color:#fff

KV cache reuse as the user types — the key optimization

The cache survives from one keystroke to the next, editing invalidates only part of it, and the saving it produces is measured in accelerators rather than milliseconds.

Every keystroke produces a prefix that extends the previous one rather than replacing it. Because attention is causal, the key and value vectors of tokens 0..n are unchanged by appending token n+1 (Prompt caching derived). So the cache built for the previous suggestion is still valid, and prefill has to run over the new suffix only.

The diagram traces three keystrokes and one edit:

• t=0. The composer holds “Thanks for sending”. The model prefills 4 tokens, leaving a KV cache 4 tokens long.
• t=1. The user has added ” that over. I’ll”. The model prefills 4 NEW tokens only, and the 8-token cache is simply the old one extended.
• t=2. The user adds ” review”. The model prefills 1 NEW token, and the 9-token cache carries forward everything before it.
• Then the user hits backspace past token 6. Truncate the cache to 6. Tokens 7+ are invalid — and ONLY tokens 7+ are, which is what makes editing cheap rather than catastrophic.

flowchart LR
    T1["t=0<br/>'Thanks for sending'<br/>prefill 4 tokens"] --> C1[("KV cache<br/>4 tokens")]
    C1 --> T2["t=1<br/>+' that over. I'll'<br/>prefill 4 NEW tokens"]
    T2 --> C2[("KV cache<br/>8 tokens")]
    C2 --> T3["t=2<br/>+' review'<br/>prefill 1 NEW token"]
    T3 --> C3[("KV cache<br/>9 tokens")]
    C3 --> INV["User hits backspace<br/>past token 6"]
    INV --> TRUNC["Truncate cache to 6.<br/>Tokens 7+ invalid —<br/>and ONLY tokens 7+"]

    style C1 fill:#2d6a4f,color:#fff
    style C2 fill:#2d6a4f,color:#fff
    style C3 fill:#2d6a4f,color:#fff
    style INV fill:#bc6c25,color:#fff
    style TRUNC fill:#9d0208,color:#fff

Editing is the interesting case, and it is handled by the same rule rather than a special one. A backspace or a mid-sentence cursor move invalidates the cache from the edit point forward, and only from there — the same invalidation semantics as prompt caching, for the same reason: everything before the edit still only depends on tokens before the edit. Truncate to the longest common prefix and re-prefill the rest.

Note that this is a throughput argument, not a latency argument, and saying so is the mark of someone who has actually measured it. Prefilling 700 tokens on a 300M model costs about 1.4 ms, so the cache saves roughly 1.4 ms of latency, which is noise against a 100 ms budget. What it saves is 696 accelerators of fleet-wide prefill, from the arithmetic above. In latency terms the cache matters mostly on device, where compute is around 200 times scarcer than on a server accelerator.

The cache is why the model runs on the device

The deployment decision comes out of a memory calculation, not the privacy argument you might expect.

Server-side caching runs into a wall that has nothing to do with compute. The wall is memory, and the argument runs in four steps: how long a session lasts, how many are open at once, how much memory each one holds, and how that compares to the memory the arithmetic itself needs. QPS below is queries per second, the count of requests the fleet handles each second.

a 340-char message at 3.3 chars/s   =  103 s of composing
50M x 6 composes x 103 s / 86,400   =  358k concurrent composers, 1.07M at peak

KV per session   =  700 tokens x 24 KB           =  17.2 MB
1.07M concurrent composers                       =  18.5 TB
accelerator memory 80 GB                         =  231 accelerators, holding cache only

versus decode:  peak 500k QPS x 6 tokens = 3.0M tok/s
                150,800 tok/s per accelerator at batch 64
                                                 =  20 accelerators

Each line unpacked, because the units are where this kind of estimate usually goes wrong:

• 103 s of composing. A 340-character message typed at the average 3.3 chars/s takes 340 / 3.3 = 103 seconds. That is how long one KV cache has to stay resident.
• 358k concurrent composers. Total composing-seconds per day is 50e6 users x 6 composes x 103 s = 3.09e10 seconds. A day is 86,400 seconds, so on average 3.09e10 / 86,400 = 358,000 people are composing at any instant. The 1.07M is that figure at the 3x peak factor derived under Scale.
• 17.2 MB per session. 700 tokens x 24,576 bytes/token = 17.2e6 bytes.
• 18.5 TB. 1.07e6 sessions x 17.2e6 bytes = 1.84e13 bytes, which is 18.4 TB — call it 18.5.
• 231 accelerators. 1.84e13 / 80e9 bytes per accelerator = 230. These accelerators do no arithmetic whatsoever. They exist to hold bytes.
• 20 accelerators for decode. Peak load is 500,000 QPS x 6 tokens = 3.0e6 tokens/s. One accelerator produces 64 tokens per 0.425 ms step, which is 64 / 0.000425 = 150,800 tokens/s. So 3.0e6 / 150,800 = 20.

Cache residency costs 231 / 20 = 11 times more hardware than the actual computation.

There are three ways out:

1. Evict caches on a short inactivity TTL. A TTL is a time-to-live: the cache is discarded automatically if the user has been idle that long. You then pay for re-prefills when they come back.
2. Page the cache out to the host machine’s ordinary memory. That means moving 17.2 MB over PCIe, the bus connecting an accelerator to its host, at 64 GB/s. 17.2e6 / 64e9 = 0.27 ms, which is affordable against a 100 ms budget.
3. Put the model where the cache is naturally free.

That third option is the design. On the device there is exactly one session: 17.2 MB of cache is nothing at all, and it lives next to the only user who is allowed to see it.

The memory arithmetic, not the privacy argument, is what forces the on-device deployment. The privacy benefit arrives as a second reason rather than the first. Volunteering that ordering is worth points, because it shows the decision was derived rather than assumed.

On-device versus server, decomposed

Compared end to end, the two deployments are separated not by the typical case but by the tail.

The two columns below itemise every millisecond from keystroke to pixels. Read down each column, add it up, and then compare the last two lines rather than the totals — that is where the decision is.

ON DEVICE                                    SERVER
keystroke -> debounce        40.0 ms         keystroke -> debounce         40.0 ms
prefill 4 new tokens          1.5 ms         client -> edge                18.0 ms
decode 6 tok @ 4.41 ms       26.5 ms         queue + batch admission        5.0 ms
render                        2.0 ms         prefill 4 new tokens           0.2 ms
                            --------         decode 6 tok @ 0.42 ms         2.5 ms
                    p50 =    70.0 ms         edge -> client                16.0 ms
                    p99 =    78 ms           render                         2.0 ms
                    (thermal throttling)                                  --------
                                                                  p50 =   83.7 ms
                                                                  p99 =  ~210 ms
                                                                  (network tail)

The debounce line is the 40 ms of quiet the debounce policy requires. It is the same 40 in both columns because it is a client-side wait that happens before either design does any work.

Both p50 figures are just the column summed: 40 + 1.5 + 26.5 + 2 = 70.0 on device, 40 + 18 + 5 + 0.2 + 2.5 + 16 + 2 = 83.7 on the server.

The p50 difference is 13.7 ms, and it would not decide anything on its own. Both medians sit comfortably under 100 ms.

The p99 difference is 210 - 78 = 132 ms, and p99 is the budget — because the suggestion that misses the deadline is the one that renders after the user has already typed past it.

Note also why each tail is what it is, because the two causes have different fixes:

• The device’s p99 is set by thermal throttling — the phone slowing itself down when it gets hot. It is bounded and it is yours to manage.
• The server’s p99 is set by the network tail — the small fraction of round trips that take far longer than the median. No amount of server capacity fixes it, because the delay is not on your hardware.

The phone’s memory bus caps the model size

Once you have decided to run on the device, the device’s memory bandwidth is what sets how large the model can be. A phone’s memory runs at roughly 68 GB/s — LPDDR5X is the low-power memory standard current phones use — against the 3.3 TB/s of a server accelerator. That is about 48x less.

Decode is bound by exactly that number, so divide weight bytes by 68 GB/s:

300M int8 (300 MB):  4.41 ms/token  ->  6 tokens = 26.5 ms   fits
1B   int8 (1.0 GB): 14.7  ms/token  ->  6 tokens = 88.2 ms   does not fit

Checking the first row: 3e8 bytes / 68e9 bytes-per-second = 4.41e-3 s, and 6 x 4.41 = 26.5 ms. The second row is the same division with 1.0 GB, giving 14.7 ms per token and 88.2 ms for six — which alone exceeds a budget that also has to pay for debounce and render.

The phone’s memory bus, not the accuracy curve, is what caps the model at a few hundred million parameters.

A server tier still earns its place for exactly one case: the first suggestion after the composer opens on a long reply thread, where 4,000 tokens of thread context materially help and the device would rather not prefill them. The user has just clicked reply and is not yet typing, so the deadline there is 400 ms rather than 100 — which is the latency assumption relaxing in the one moment where it genuinely does not apply.

Assumptions in this stage.

Load-bearing: that the user’s device can run a 300M int8 model at all. The entire deployment rests on a phone with a neural accelerator and ~68 GB/s of memory bandwidth.

But the fleet is not uniform. A five-year-old low-end handset has a fraction of that bandwidth, so a 26.5 ms decode becomes 80 ms and the p99 blows the budget — on precisely the devices where it is hardest to notice from a fleet-wide dashboard.

What breaks, and the fallback: segment the latency guardrail by device class, and on devices below the bar fall back to the n-gram model, which is microseconds everywhere.

Also load-bearing: the 103-second composing session used in the concurrency arithmetic, which is a 340-character message at average typing speed. Shorter messages mean more sessions per second but less cache resident per session, and the two effects partly cancel. The number that does not cancel is peak concurrency, and it is worth measuring rather than assuming.

Debouncing and cancellation

Two client-side policies decide when a request is issued at all and what happens to results that arrive too late — and the second is the difference between a feature that feels solid and one that feels broken.

Firing the model on every keystroke is both wasteful and worse for the user. Debouncing means waiting for a short pause before acting on input, so that a burst of fast keystrokes produces one request instead of ten:

This table is where the “48 evaluations per compose” figure from the very first table comes from. At 14 requests per 100 keystrokes, a 340-character message produces 340 x 0.14 = 47.6 model evaluations — call it 48. Every traffic number in the chapter, from the 500k peak QPS to the 288 daily evaluations per user, is downstream of that one multiplication.

The 40 ms threshold does not come from the mean, and the mean is what makes it look pointless. At 6.7 characters per second the average gap between keystrokes is 150 ms, and a 40 ms gate applied to a population of 150 ms gaps would pass essentially everything — it would be a gate that does nothing. But the table two rows above says it cuts word-boundary requests from 22 per 100 keystrokes to 14, and that is a measurement, not a prediction: 8/22 = 36% of the gaps at word boundaries are shorter than 40 ms.

A mean of 150 ms cannot produce a 36% mass below 40 ms. Only burstiness can, and burstiness is the actual claim.

Real typing is not a steady drip at a fixed rate. It is runs of practised muscle memory — a familiar word, a common phrase, a sign-off typed ten thousand times — separated by genuine pauses for thought, word choice, and re-reading.

Within a run, successive keystrokes land 20-30 ms apart, far faster than the average suggests. Between runs, gaps stretch to several hundred milliseconds.

So the distribution is bimodal, and the 150 ms average sits in the trough between the two modes, where comparatively few gaps actually fall. The mean describes no keystroke in particular.

That trough is what the gate is selecting on. 40 ms is placed inside it — above the intra-burst mode, below the inter-burst one — so that it separates the two populations rather than trimming a tail.

It is a threshold on burst membership, not on elapsed time as such:

• The 36% it removes are mid-burst keystrokes — positions where the user is still executing a word they had already decided on, and a suggestion has nothing to offer.
• The 64% it passes are the positions where typing actually paused — the only place a completion is worth reading.

Derive the number from your own inter-keystroke histogram, not from the mean. The mean is the one statistic of this distribution that carries no information.

Cancellation is not optional. Every keystroke bumps the sequence number; a result carrying a stale sequence number is discarded on the client, and any in-flight server request is cancelled. Without this you ship the worst bug this feature has — a suggestion that fades in after the user has typed past it, forcing them to re-read text they already wrote.

Metrics

Some of this system can be measured before launch and some only after — and the metric everyone reaches for first, acceptance rate, is the one that will mislead you.

Offline

These are the numbers computed on held-out sent mail, before anything ships. Note that only one of them is a headline; the rest are gates and diagnostics.

Exact match is the right metric here, and it is the one case in this whole folder where that is true (2a there is no single correct label so evaluation is the system).

Generative systems usually have many acceptable outputs, which is why exact string comparison is normally useless as a measure. Here the acceptable set has size one: the user gets a single Tab key, so a suggestion is useful only if it is a literal prefix of what they were going to type. A paraphrase scores zero and deserves zero.

BLEU and ROUGE — the two standard overlap-based text metrics, which score a candidate by how many n-grams it shares with a reference — are actively wrong here, for two reasons. A 6-token span has no room for n-gram overlap to mean anything. And partial semantic credit is credit for something the product cannot cash: there is no “70% accept” key.

Counterfactual replay

The counterfactual replay is the offline harness that comes closest to the online number. A counterfactual measurement asks what would have happened under a policy that was never actually run.

The procedure: take held-out sent messages, walk every position, run the real trigger and length policy at each one, and count the characters that would have been saved had the user accepted every suggestion that exactly prefixes what they in fact typed.

It is the only offline number that tracks the online one, because it exercises the decision layer and not just the language model.

It is also systematically optimistic — and, this is the good part, the size of that optimism is not a fudge factor but a parameter you have already estimated:

simulated chars/day    566      <- assumes every exactly-correct suggestion is accepted
observed  chars/day    316
ratio                 0.56      =  A_ok = 0.55, the measured P(accept | correct)

The simulator’s optimism IS A_ok — the replay assumes every correct suggestion is accepted, and in reality only 55% are. So a change in the gap between simulation and reality is a signal that user behaviour moved, not that the simulator broke. Calibrate the replay once against a live A/B test, then trust it as a release gate and never as a forecast.

Online

What is the feature actually worth per user per day? Derive it, then price the same day in utiles — which shows where nearly half of that value goes.

The headline is not acceptance rate. It is characters saved per user per day, and every input to it has already been derived somewhere above.

evaluations/user/day   =  6 composes x 48 evaluations            =  288
suggestions shown      =  288 x 0.14 coverage                    =   40.3
mean calibrated q among shown (threshold 0.633)                  =    0.78
acceptance             =  0.78 x 0.55  +  0.22 x 0.03            =    0.436
accepts/day            =  40.3 x 0.436                           =   17.6
chars saved/day        =  17.6 x 18                              =  316

against 6 x 340 = 2,040 chars typed/day  ->  15.5% of typing eliminated
                                         ->  ~95 seconds/user/day at 3.3 chars/s

The acceptance line is the only one that is not a plain multiplication, so read it slowly. Of the suggestions shown, a fraction 0.78 are correct and get accepted 55% of the time (A_ok); the remaining 0.22 are wrong and get accepted 3% of the time anyway (A_wrong). So 0.78 x 0.55 = 0.429, plus 0.22 x 0.03 = 0.0066, gives 0.436.

The last two lines: 316 / 2,040 = 15.5% of characters never typed, and 316 / 3.3 chars-per-second = 96 seconds of typing avoided per user per day.

Acceptance rate is derived here, not assumed. It falls out of the mean calibrated q among shown suggestions, mixed through A_ok and A_wrong. If your reported acceptance rate does not reconcile with your threshold and your calibrator, one of the three is wrong.

The same day, priced in utiles

The 316 characters is the gross number. Now run the same day through the trigger’s own cost model, which also charges you for the suggestions that went wrong.

gain          17.6 accepts x 1.00                        =  +17.6
dismissals    22.7 shown-and-dismissed x 0.05            =   -1.1
bad accepts   40.3 x 0.22 wrong x 0.03 accepted x 30     =   -8.0
                                                            ------
                                                             +8.5

The middle line’s 22.7 is just 40.3 shown - 17.6 accepted. The bottom line works out to 40.3 x 0.22 = 8.9 wrong suggestions shown per day, of which 8.9 x 0.03 = 0.27 are accepted anyway — about one every four days — and each costs 30.

Bad accepts consume 8.0 / 17.6 = 45% of the gross value of the feature, at a rate of one every four days per user.

That single line is the whole argument for a precision-first design, and it is why the threshold lands at 0.633 rather than 0.05. The threshold derivation and this product accounting were done independently and they agree, which is the main reason to trust either of them.

Assumptions in this stage.

Load-bearing: that acceptance is derived rather than measured and set. Every figure in the ledger — 40.3 shown, 17.6 accepted, 316 characters — follows from four inputs: coverage, the mean calibrated q among shown suggestions, A_ok, and A_wrong.

If the measured acceptance rate does not reconcile with those four inputs, do not adjust the ledger. One of the calibrator, the threshold and the behavioural rates is wrong, and the reconciliation failure is the most valuable signal the system produces.

State out loud: 48 evaluations per compose and 6 composes per day, which is what turns coverage into suggestions shown.

Why click-through-style metrics mislead here

Acceptance rate — the metric every stakeholder will ask for first — is not merely imperfect but actively misleading, in three separate ways.

Acceptance rate is accepts divided by suggestions shown. It is the same shape as CTR, click-through rate, the standard measure in search and advertising: the fraction of things shown that were clicked. It fails as a headline here for three distinct reasons.

1. You control the denominator. Raise the threshold and acceptance rate rises while the product gets worse. The table below runs the same daily ledger at three thresholds. Compare the Acceptance column with the Chars saved/day column; they move in opposite directions.

Raising the threshold restricts you to the positions where the model is most confident, so the suggestions you do show are better. But you show far fewer of them, so the user saves far fewer characters.

Acceptance rate improves by 52.4/43.6 = 20% while the product gets (316-68)/316 = 79% worse. Any metric whose denominator is a parameter you tune is a metric you will tune — and here the tuning knob is one line of config.

2. There is no impression the user opted into. A web result is shown because the user scanned a list; a Smart Compose suggestion appears unbidden in the middle of their sentence. A non-click is not disinterest — it is often cost, a fixation spent on text that turned out to be wrong.

3. A rejection has a negative price, and click-through rate counts it as zero. That is exactly the C_r term in the trigger derivation. A click-through framing prices a rejection at nothing and therefore recommends showing far more suggestions than the utility calculation permits.

Report the pair — characters saved and suggestions shown — or report the single utility number that already prices both.

Guardrails that must not move

A guardrail is a metric that does not have to improve for a launch to go ahead, but whose regression blocks it regardless of how good the headline looks. These five are the ones that catch the failures characters-saved cannot see.

Retraction rate is the metric to volunteer. Acceptance says the user pressed Tab; retraction says whether they meant it. It costs nothing to instrument and it is the only direct read on A_wrong, which is the term the entire threshold derivation hinges on.

The A/B test

Sizing the experiment leads somewhere unexpected: the binding constraint is calendar time rather than users.

An A/B test randomly assigns users to a control arm and a treatment arm and compares outcomes. Randomize on the user, never on the individual request: the effects that matter here — habituation, trust, and the decision to disable the feature — accumulate per person, and randomizing per request understates the true variance and therefore manufactures statistically significant results that do not replicate (Ab testing).

The sample size follows the standard formula n ≈ 16 σ² / δ², in which:

• delta (δ) is the effect you want to be able to detect.
• sigma (σ) is the per-user standard deviation — how spread out the per-user outcomes are. It is large here because a few users compose vastly more mail than the rest.
• n is the users needed in each arm.

detect a 4% relative change in chars saved/day:  delta = 0.04 x 316  =  12.6 chars
per-user sd (heavy right tail)               :  sigma = 350

n per arm  ≈  16 sigma^2 / delta^2  =  16 x 122,500 / 159  =  12,300 users
with CUPED on pre-period typing volume (~40% variance reduction)  ->  7,400

Substituting: sigma^2 = 350^2 = 122,500 and delta^2 = 12.6^2 = 159, so 16 x 122,500 / 159 = 12,300.

CUPED is a standard variance-reduction technique — controlled experiment using pre-experiment data. It subtracts off each user’s own pre-experiment behaviour, here their typing volume, so the comparison is not swamped by the fact that some people simply write more email than others. It cuts the required sample by about 40%, giving 12,300 x 0.6 = 7,400.

Both numbers are trivial against 50M daily active users, which means the binding constraint is not sample size but time: read the experiment at week 3, not week 1, because a new inline suggestion gets tried out before it gets used, and the week-1 number measures novelty.

Assumptions in this stage. Load-bearing: that user behaviour is stable within the measurement window. The whole cost model assumes A_ok and A_wrong are properties of users rather than of how long they have had the feature, and the week-1-versus-week-3 gap is direct evidence that this is false early in a user’s exposure. What breaks: an experiment read too early over-reports both acceptance and annoyance, which pushes the threshold in an arbitrary direction. The fix is procedural, not statistical — fix the read window before the experiment starts.

Scale and cost

Price the fleet under each deployment option and one conclusion dominates: renting a model through a hosted API — an application programming interface, meaning you send text over the network to somebody else’s model and pay per token — loses twice over, once on latency, and then again, separately, by three orders of magnitude on cost.

Start with the request rate, since every dollar figure below is that number times a unit price.

50M DAU x 6 composes x 48 evaluations   =  14.4B requests/day
                                           167,000 QPS average, ~500,000 peak

The average QPS is 1.44e10 / 86,400 seconds = 167,000. The peak is that at a 3x peak factor — traffic is not flat across a day — which is the 500,000 used in every capacity estimate above.

Now price four deployments against that load. Two terms in the table:

• MTok means a million tokens, the unit hosted model providers bill in. The two figures separated by a slash are the input and output prices, so $5/$25 is $5 per million input tokens and $25 per million output tokens.
• NPU means neural processing unit, the dedicated accelerator on a modern phone that runs models without waking the main processor.

The hosted-API rows are one substitution each, and it is worth doing once by hand because the magnitude is easy to disbelieve. Each request sends 700 tokens in and gets 6 out, at 14.4B requests a day:

input cost per request   =  700 tokens / 1,000,000 x $5   =  $0.0035
output cost per request  =    6 tokens / 1,000,000 x $25  =  $0.00015
                                                             ---------
total per request                                            $0.00365

x 1.44e10 requests/day  =  $52.6M/day  =  $19.2B/year

Note where the money is: the input is 96% of the bill. You pay to re-send the same growing prefix 48 times per compose, which is the exact work the on-device KV cache does for free.

The API path is disqualified twice over — first by a 36 ms decode step that blows the budget before the network is involved, then by a bill three orders of magnitude above the alternative.

Say it in that order, because the latency argument is the one that cannot be engineered around. Note also that naming the tier is load-bearing: the frontier model you would actually want is 5x the price of the cheap one, and the cheap one is not obviously better than the 300M you trained for this distribution.

Battery, the cost a reviewer will raise

Battery is the real cost of the on-device design. Price it the same way as everything else:

288 inferences/day x ~28 ms of NPU each  =  8.1 seconds of NPU per day

That 28 ms is the on-device prefill plus decode from the comparison above (1.5 + 26.5). Eight seconds a day is negligible against how long the screen is on.

The number that would not be negligible is a 1B model at 88 ms per inference: 288 x 88.2 ms = 25 seconds a day, and a measurably warm phone. That is the same memory-bandwidth constraint that already capped the model size, arriving a second time in a different currency.

Assumptions in this stage.

Load-bearing: the peak factor that turns 167,000 average requests per second into 500,000 at peak. Every accelerator count in the table is sized against the peak, not the average. A heavier peak — one time zone dominating the user base, or a workday that starts at the same hour for everyone — moves the server fallback’s bill proportionally, without changing anything on the device path.

State out loud: $2.50 per GPU-hour fully loaded, and 300 TFLOP/s effective per accelerator.

What does not depend on any of it: the on-device design costs $0 marginal at any traffic level. That is the deeper reason it wins — it is the only option whose bill does not scale with success.

Failure modes

Seven concrete traces of the system going wrong, each with the mechanism that produces it and the specific control that stops it. The last one is the one nobody volunteers.

1. A factually wrong completion in a professional email

The trace below is the model completing a number it has no way of knowing. Watch the confidence figure — 0.71 is high, and it is high for the wrong reason.

composed:   "Hi Dana — confirming the invoice total is $"
model top-1: "1,200.00 and payment is due within 30 days."   p = 0.71
             (learned from thousands of near-identical invoice emails)
user Tabs (autopilot)
sent:       "...confirming the invoice total is $1,200.00 and payment is
             due within 30 days."
actual invoice: $12,000.00, net 15

Nothing in the model can know the total. Detection: retraction rate spikes on messages containing currency tokens. Guard: hard suppression inside numeric, currency, date, and time tokens — a rule in code, not a prompt, because the model’s confidence here is high and completely uninformative.

2. An inappropriate or biased completion

composed:   "I met with the cardiologist this morning and I'll ask "
model top-1: "him to send over the notes."                    p = 0.68

The model inferred a gender from a job title, because the corpus co-occurrence says so. Guard: the derived pronoun suppression above (q* = 0.992, unattainable). The point to make in the interview is that this is not a special case bolted on — it is the same expected-utility rule with a different C_w. A design that reaches the right answer by the same route it reaches every other answer is a design that will also handle the class you have not thought of.

The related trace: a protected-attribute term near the cursor turns the suggestion into a stereotype completion. Same treatment, suppression by proximity rule.

3. Memorized personal data from another user

Personally identifiable information (PII) is any data that can be traced to a specific individual — an account number, a routing number, a home address.

composed:   "For the wire, the routing number is "
model top-1: "021000021 and the account is 4417"              p = 0.44

The model saw a real routing number during training and is reproducing it. Detection: the canary suite, which measures exposure at controlled frequencies before the model ships. Guards, in layers: differential privacy in training bounds any single user’s influence; the k-anonymity whitelist blocks any suggestion string not seen across 50+ users; the numeric suppression rule blocks this specific shape regardless.

4. The stale suggestion race

This is a timeline, in milliseconds, of a request that arrives 12 ms after the prefix it was computed for stopped existing. The seq column is the sequence number described under Serving.

t=0    ms  user types "...I'll rev"          seq=41  request fires
t=62   ms  user types "i"                    seq=42  seq 41 now stale
t=74   ms  response for seq=41 arrives:      "iew it and send it back"
           -> rendered without a seq check
t=76   ms  grey text appears after "revi", offset by one character
t=140  ms  user types "e", suggestion vanishes

The user read text that was never valid. Guard: compare the sequence number on arrival and discard mismatches; cancel in-flight requests on keystroke. This is a five-line fix and its absence is the most common way the feature feels broken.

5. Cross-session cache reuse

The bug is on the first line of the trace below: a cache key built from the prefix alone, with nothing identifying whose prefix it is.

cache key = sha256(prefix_tokens)                 <- WRONG

user A composes:  "Hi,"                 -> cache entry X written
user B composes:  "Hi,"                 -> cache HIT on X
                                        -> B's suggestion is conditioned on
                                           A's warm state, and on a longer
                                           prefix the collision leaks content

Two different users’ prefixes hash to the same value constantly here, because email openings are formulaic — sha256 is a hash function, which turns text into a fixed-length fingerprint, and identical text produces an identical fingerprint by design. Guard: key the cache on (session_id, prefix_hash) rather than the prefix alone, and scope every entry to the lifetime of its session. On the device this failure cannot occur at all, which is a third independent argument for the on-device design.

6. Non-Latin scripts and code-switching degrade silently

Code-switching is moving between two languages within one message, which is ordinary behaviour for a large share of the world’s email.

The tokenizer — the component that chops text into the model’s vocabulary units — fragments non-Latin scripts far more aggressively than English (Tokens). So a 6-token suggestion might carry 2 words instead of 5.

That is what makes the failure silent: characters saved per suggestion halves, while acceptance rate looks unchanged, so the aggregate dashboard shows nothing at all.

Guard: segment every metric by language and by script, and set the length policy for each language from its own q(L) curve rather than from a global one.

7. The feature eats its own training data

This is the closed loop, and it runs over quarters rather than milliseconds. Follow it round once:

1. Users accept suggestions, so sent mail becomes more templated.
2. The next training round sees a more templated corpus, so suggestions get blander.
3. Acceptance rate rises, because bland suggestions are safe ones.
4. But the value per accepted suggestion falls, because a suggestion the user would have written anyway saves less that matters.

Every number on the dashboard improves while the product decays. It is exactly the shape of drift described in Detecting drift without labels the real production problem.

Guard: hold out a permanent control cohort who are never shown suggestions, and treat their sent mail as the reference distribution. Then monitor the KL divergence — Kullback-Leibler divergence, a standard measure of how far one probability distribution has moved from another, zero when they are identical and growing as they separate — between the treatment cohort’s and the control cohort’s sent-mail n-gram distributions. If it grows monotonically over quarters, the loop is closing. This is the failure mode nobody mentions unprompted, and it is worth 5 minutes of the interview.

Alternatives considered and rejected

Every design below is one a reasonable person would propose instead, paired with the specific number that kills it — each rejection quantified rather than asserted.

Three names in it are worth glossing first.

• seq2seq is short for sequence-to-sequence, the encoder-decoder framing already argued against above.
• KenLM is a widely used implementation of an n-gram language model, which predicts the next word purely from counts of how often each short run of words was followed by each other word. No neural network, microseconds per query, and no ability to generalise beyond runs it has literally seen.
• A semantic cache stores past requests keyed by meaning rather than by exact text, so a new request that is merely similar to an old one gets the old answer back. It is a large win when generation is expensive, and a liability when generation is cheap.

Interviewer pushback

Eleven questions this design attracts, what each one is really testing, and the answer that satisfies it. Use them as a self-test: if you can produce each answer from the derivations above without looking, you have the chapter.

“Why not just call a good hosted model?” Testing: whether latency is real to you or a slogan. Because the decode arithmetic rules it out before cost does. A 7B at batch 64 with 700-token contexts is a 6 ms decode step, so 6 tokens is 36 ms — that is the whole decode budget, and I still owe 34 ms of network and 40 ms of debounce. A 70B is 281 ms, 2.8x the entire end-to-end budget. Then, separately, the bill: $19.2B a year at frontier pricing, or $3.8B if I rent the cheapest tier instead, against $5.6M self-hosted and $0 on device. The latency argument is the one that cannot be engineered around.

“A 300M model is much worse than a 7B. How do you justify it?” Testing: whether you know what selective prediction buys. In aggregate it disagrees with the 7B on 29% of next tokens. But the trigger only fires on the confident 12% of positions, and on that slice agreement with the 7B is 96%. The shipped disagreement is 4 points, not 29. The model is allowed to be bad as long as it knows when — and that is the same reason the optimal suggestion length jumps from 2 tokens unconditionally to 5-6 conditional on firing.

“How do you set the threshold for showing a suggestion?” Testing: whether thresholds are derived or tuned. From the cost asymmetry. A correct suggestion is accepted 55% of the time and worth 1 utile; a wrong one costs 0.05 in attention and, 3% of the time, gets accepted anyway at a cost I price at 30. That gives C_fp = 0.95, C_fn = 0.55, and q* = 0.95/1.50 = 0.633 — the standard cost-matrix threshold. Worth noting the counterfactual: on typing time alone, 0.05/(0.05 + 0.55), the threshold would be 0.083 — so the bad-accept term is moving it by a factor of 7.6. And q has to be calibrated, because the raw sequence probability is length-confounded.

“Your acceptance rate went from 44% to 52%. Ship it?” Testing: whether you take a good number at face value. It is a trap. No — that is the signature of a raised threshold, not a better model. Coverage almost certainly fell, and if it fell from 14% to 2.5% then characters saved went from 316 a day to 68: acceptance improved 20% while the product got 79% worse. Acceptance rate has a denominator I control, and the knob is one line of config. I would look at characters saved per user per day, with suggestions shown as the guardrail, and check that retraction rate has not moved.

“Why does the KV cache matter if prefill is only a millisecond?” Testing: whether you can tell a latency argument from a throughput argument. It does not matter for latency, and I would say so. On a 300M model, prefilling 700 tokens is about 1.4 ms, so the cache saves noise. What it saves is fleet compute: at 500k peak QPS, re-encoding 700 tokens per request is 2.1e17 FLOP/s — 700 accelerators at 300 TFLOP/s — versus four when you prefill only the 4 new tokens. The cache is a 175x cost lever and a 0x latency lever, and getting that backwards is a common mistake.

“So why on-device, if the server has more compute?” Testing: whether the on-device argument is reasoned or reflexive. Two reasons and privacy is the second one. First, memory: 700 tokens of KV is 17.2 MB per session, and a 340-character message takes 103 seconds to compose, so peak concurrency is about 1.07M sessions — 18.5 TB, which is 231 accelerators holding cache against 20 doing the arithmetic. Cache residency costs 11x more hardware than the computation. Second, p99: server p50 is about 84 ms and p99 is around 210 ms because of the network tail, against a 100 ms budget — and p99 is the budget, since the suggestion that misses is the one that renders after the user has typed past it. On device there is exactly one session, the cache is free, and there is no network.

“What stops it from completing someone else’s credit card number?” Testing: whether you know memorization is a measurable property. Four layers, and the first one is the measurement. Canary insertion: inject synthetic secrets at known frequencies before training and measure exposure afterwards — without DP, canaries at frequency 8 are extractable at 14 bits. Then DP-SGD — differentially private stochastic gradient descent, meaning ordinary training with per-client clipping plus Gaussian noise on the aggregate — which bounds any one user’s influence and drops that canary below the detection floor. Then a k-anonymity whitelist: no suggestion string ships unless it appears in 50+ distinct users’ mail, which is a filter in code rather than a property of the model. Then the numeric suppression rule, which blocks that shape regardless of what the model wanted.

“Would you suggest pronouns?” Testing: whether a values question gets an engineering answer. No, and it falls out of the same formula as everything else. A pronoun saves 3 characters, so the upside is 0.17 utiles. A pronoun error is a personal misattribution in a message the user then sends; price that at 400 and the required confidence is 12.05/12.14 = 0.992. A calibrated 99.2% on a demographic inference the model is making from a job title is not achievable, and I would not trust the number if it said otherwise. When the derived threshold exceeds what calibration can certify, you suppress the class.

“How do you evaluate this offline when there is no single right answer?” Testing: whether you apply the framework’s rule or recite it. This is the exception where there is one: the user gets a single Tab, so a suggestion is useful only if it is a literal prefix of what they were about to type. A paraphrase deserves zero. So exact-prefix match is right, and BLEU or ROUGE would be actively wrong on a 6-token span. The headline offline number is a counterfactual replay over held-out sent mail — walk every position, run the real trigger and length policy, count characters that would have been saved. It runs 0.56x optimistic against live behavior, and that factor is not a fudge — it is A_ok, the measured probability that a correct suggestion is actually accepted. So I calibrate it once against an A/B and then use the replay as a gate, never as a forecast.

“What is this feature actually worth?” Testing: intellectual honesty, and whether your numbers connect. About 316 characters a day per user against 2,040 typed, so 15.5% of typing, or 95 seconds. But the honest version is the utility ledger, not the gross: 17.6 accepts at +1, minus 22.7 dismissals at -0.05, minus one bad accept every four days at -30. Bad accepts eat 45% of the gross value. That is the number that governs the design, and it is why the threshold derivation lands at 0.633 instead of 0.05. The two calculations are independent and they agree, which is the main reason I trust either.

“You have 50M users and a week. What ships first?” Testing: whether you can sequence. The n-gram fallback plus the hard suppression rules and the debounce, on device, at a deliberately high threshold — coverage 5%, not 14%. It saves fewer characters and it cannot embarrass anyone, and its real job is to produce the first honest A_ok and A_wrong measurements, which are the two numbers the entire threshold derivation depends on and which I currently only have as estimates. Everything after that is tightening the threshold with data instead of assumptions.

The assumption ledger

Every assumption the chapter has leaned on, collected in one place, so you can state the design’s foundations in twenty seconds and say what replaces the design when each one fails. Sort each into one of three bins: state it (you are free to pick, and being wrong costs a re-derivation), ask it (the answer changes the architecture, so it is worth an interviewer’s time), and load-bearing (if it is wrong the design is not suboptimal, it is invalid) — the same three bins used in ch 01.

The sentence that makes this visible to an interviewer: “This design rests on three things. One, the deadline is 100 ms because that is a fast typist’s gap between keystrokes — relax that and I would not build this system. Two, users accept wrong suggestions about 3% of the time, which is what moves the threshold from 0.08 to 0.63; it is the number I would measure first and it is currently an estimate. Three, there are tens of millions of users, which is what makes differential privacy and the k-anonymity whitelist affordable at all.”

Cheat sheet

The one-line answers to the questions this design is most often asked. Everything here is derived above; this is the recall test.

Next: 03 — Machine Translation — where the output space is narrow enough that reference-based metrics come back to life, and the hard problem moves to the tail of the language distribution.