03 — Machine Translation

“Design machine translation for 100 languages at a billion requests a day.”

Machine translation (MT) is the task of taking text written in one language and producing text in another that means the same thing.

At 100 languages and a billion requests a day, three things decide the design, and none of them is the model architecture:

1. How you manufacture training data for the language pairs that have none.
2. How the tokenizer silently sets both cost and quality for exactly the languages you are worst at.
3. Where the router draws the line between a small fast model and a large slow one.

By the end you will be able to derive the fleet size and the daily bill from a traffic figure, explain why the industry-standard quality score ranks a factual error above a correct paraphrase, and say which assumptions the whole design would collapse without.

Vocabulary you need before the first section

These terms are used constantly below. Each is defined once, here, so that nothing later has to stop and explain it.

The instinct this prompt provokes is to start talking about transformer architecture. The actual answer is about data acquisition, a tokenizer decision made a year earlier, and which lane the router sends a request to.

Problem framing

Four things anchor everything that follows: what goes in, what comes out, the constraints every later number is derived from, and the one failure mode the entire design exists to prevent.

• Input: a span of text, a target language, and — sometimes — a declared source language.
• Output: text in the target language that means what the source meant.
• Constraints: p95 under 300 ms on the interactive surface, 100 languages, and a traffic distribution where the top 50 directed pairs are 80% of volume and the remaining 9,850 are a long tail nobody can afford to serve well one at a time.
• Why it’s hard: the failure that matters is not a clumsy sentence. It is a fluent sentence that asserts something the source did not, because no downstream reader who lacks the source language can detect it.

Say the input and the output in one sentence before going any further, because everything downstream is shaped by them: a paragraph of text plus a target language goes in, and a paragraph of text in that target language comes out. Nothing about that shape tells you which of the 9,900 possible directions the request is in, how much data exists for it, or how long the answer is allowed to take — and those three questions are the whole design.

First thing to say: “Three things decide this system and none of them is the model architecture. The data acquisition pipeline, because 9,800 of my 9,900 language directions have no parallel corpus. The tokenizer, because it silently sets both cost and quality for exactly the languages I am worst at. And the routing threshold into the big model, because at this scale 10% of traffic will be 80% of the fleet.”

Three reframes

Each reframe below names an instinct that sounds sensible and the mechanical fact that overturns it. Three pieces of vocabulary appear here for the first time; each is derived in full later, so read these as promissory notes.

• Back-translation is the trick of manufacturing training pairs by running a translation model backwards over text you already have. Derived in back-translation, derived.
• BLEU (bilingual evaluation understudy) is the standard automatic translation score. It counts how many short word sequences a candidate translation shares with a human reference. Taken apart in BLEU, and exactly what it cannot see.
• Weight residency is the fact that a model can only answer a request if its weights are sitting in accelerator memory at that moment. You rent that memory by the hour whether or not a request arrives, which is why the cost does not amortize — meaning it is not spread thinner as traffic grows. Priced in the pairwise fleet dies on residency.

In each row, the middle column is what a candidate says first, and the right column is the fact that makes it wrong.

Assumptions in this stage. Every section of this chapter ends with a block like this one. It sorts the section’s assumptions into three bins: things you state (you are free to pick, and being wrong costs a re-derivation), things you ask (the answer changes the architecture), and things that are load-bearing (if one is wrong, the design is not suboptimal — it is invalid).

• State out loud: 100 languages, 1.2 billion requests a day, and an average request of about 60 source tokens. Every fleet number later in the chapter is a product of those three.
• Ask, never assume: the traffic distribution across pairs. “Top 50 pairs are 80% of volume” is the sentence that makes a single multilingual model the right answer and a per-pair fleet the wrong one. If traffic were uniform across 9,900 directions, the residency argument below would be far weaker.
• Load-bearing — the 300 ms p95 budget on the interactive surface. It is what forces the two-lane split. Relax it to two seconds and the whole fleet collapses into one large model.
• Load-bearing — a fluent falsehood is the expensive failure. That is what makes the serving-time hallucination guards mandatory rather than nice to have. If a wrong translation were merely mildly annoying, you would ship a single model with no guard layer and spend the money on language coverage instead.

ML objective

Underneath the whole system is one loss function, and two facts extracted from it turn out to explain every failure mode later in the chapter.

The system is a conditional sequence model, written p(y | x, L_tgt). Read that as: the probability of an output sentence y, given a source sentence x and a tag L_tgt naming the target language.

Training maximizes the log-likelihood of the human reference translation — the logarithm of the probability the model assigns to the words a human actually wrote. Maximizing that is the same as minimizing this loss:

loss  =  - (1/|y|) · sum over t of  log p( y_t | y_<t, x, L_tgt )

Read the expression symbol by symbol:

So: for every position in the target sentence, the model is scored on the probability it put on the correct next word. A model that puts 0.9 on the right word contributes -log(0.9) = 0.105; one that puts 0.1 on it contributes -log(0.1) = 2.303. Confidently wrong is expensive; confidently right is nearly free.

Two consequences fall straight out of that expression, and both matter later.

The loss is only over target tokens. The sum runs over t in y, never over x. Nothing in the objective penalizes ignoring the source. The decoder — the half of the model that emits the output one word at a time — is a language model that has been offered a conditioning signal from the encoder, the half that reads the source. When that signal is uninformative — an out-of-domain source, a garbled source, a language the encoder barely represents — the highest-likelihood continuation is whatever the decoder’s language model prefers on its own. That is not a bug to patch; it is the objective doing exactly what it says. Every hallucination failure mode in this chapter is a consequence of the fact that the loss lives on the target side.

Decoding is a separate decision from training. Training maximizes the likelihood of one token at a time; serving has to search for a high-scoring whole sequence. The default search is beam search: keep the k best partial translations at every step and extend all of them, rather than greedily committing to the single best next word. Scores are length-normalized so that short outputs do not win by default. Beam search introduces a failure the training objective never saw — degenerate repetition, derived in failure mode 4.

What interviewers probe: “Why not just train on more data?” Because for the pair you actually care about, more data does not exist. The interesting engineering is in what you do when the supervised signal is absent, which is what data and labels covers next.

Assumptions in this stage.

• State out loud: that the model is a single conditional sequence model with a target-language tag, rather than a separate model per direction. The alternative is priced in model choice and rejected on serving cost.
• Ask: whether document-level context is available at request time. A loss defined over a single sentence pair cannot learn anything about the sentence before it. That decision changes the input spec rather than the arithmetic.
• Load-bearing: that the loss lives on the target side and nothing forces the model to depend on the source. If that were false — if the objective contained a term rewarding source fidelity — back-translation would not work for the reason given below, hallucination under domain shift would not be the dominant failure, and the entire serving guard layer would be solving a problem that does not exist.

Data and labels

How much real training data exists? Very little — which is why the two techniques that manufacture the rest, mining translation pairs out of web crawls and back-translation, carry this part of the design.

The shape of the problem

Count the supply of real parallel data against the demand, and the gap between the two is the reason everything after it exists.

Start with the demand. Every language can be a source and every other language can be a target, so 100 languages give

100 languages × 99 possible targets each  =  9,900 directed pairs

Now the supply. Almost all published parallel data has English on one side, because that is where the translation industry has always pointed. So count the English-centric pairs — pairs with English on one side:

99 non-English languages, English -> that language   =   99
99 non-English languages, that language -> English   =   99
                                                        ----
                                                         198 directed pairs

That is the whole supply, and it is 2% of the demand. Public parallel corpora cover perhaps 100 of those 198 at useful volume, so the realistic picture is:

The row to look at is the last one. 9,702 / 9,900 = 98% of the directed pairs have essentially no real training data at all — that row is where the design lives. Counting it the other way: roughly 100 directed pairs have a usable corpus, so 9,800 do not.

Two terms in that table are used from here on. Zero-shot means translating a direction the model was never trained on. Mined bitext means translation pairs discovered inside a web crawl rather than published as a corpus.

Pipeline

The whole data factory fits in one diagram, and the two derivations that follow — mining and back-translation — each own a leg of it.

The two legs join at the bottom: the mining leg runs down the left, the back-translation leg down the right, and the mixture box is where they meet. Here is what each box does.

The mining leg — turning a web crawl into training pairs.

1. Web crawl. Raw documents, grouped by the language the crawler thinks they are in.
2. Sentence split, plus language ID on both sides. Language ID is a small classifier that guesses which language a piece of text is in. You re-run it per sentence because a crawl labelled “Icelandic” is full of English.
3. Embed every sentence. An embedding is a list of a few hundred numbers, positioned so that sentences with similar meanings land near each other. That is what lets a machine ask “are these two sentences about the same thing?” as a distance question instead of a linguistics question. Put to work in mining bitext below.
4. ANN index. Approximate nearest neighbour: a data structure that finds the closest vectors to a query without comparing against all of them. That approximation is what makes the search affordable at billions of sentences.
5. Nearest-neighbour mining. Query one language’s index with the other language’s sentences, scoring candidates with the margin criterion derived below.
6. Quality filters. A bank of reject rules that throws away 60 to 80% of the candidate pairs — discarding them, not downweighting them. What survives is a mined parallel corpus.

The back-translation leg — manufacturing pairs out of thin air. Separately, monolingual target text is fed backwards through a reverse translation model using sampled decoding. The result is synthetic pairs whose source side is noisy machine output and whose target side is real human text. That asymmetry is the subject of back-translation, derived, and it is the single most valuable idea in this section.

The mixture — deciding how often each pair appears. Mined and synthetic pairs are combined by temperature-sampled mixing: a pair p is drawn in proportion to D_p^(1/T), where D_p is how many pairs that language direction has and T is a temperature knob that flattens the mixture. T = 1 samples in natural proportion to corpus size; larger T pulls everything toward uniform. Both symbols are worked out with real numbers in the data mix.

That mixture trains the forward source-to-target model. The dotted arrow at the bottom is the loop: a trained forward model seeds a better reverse model, which produces better back-translations for round two.

flowchart TD
    CR["Web crawl<br/>· documents per language"] --> SEG["Sentence split<br/>+ language ID both sides"]
    SEG --> EMB["Multilingual sentence encoder<br/>embed every sentence"]
    EMB --> IDX[("ANN index<br/>per language")]
    IDX --> MINE["Nearest-neighbour mining<br/>margin criterion"]
    MINE --> FILT{"Quality filters<br/>langid · length ratio<br/>near-dup · MT-detector<br/>entity + numeral agreement"}
    FILT -->|"reject 60-80%"| DROP["Discard"]
    FILT -->|keep| PAR[("Mined parallel<br/>corpus")]
    MONO[("Monolingual<br/>target text")] --> BT["Reverse model T->S<br/>sampled decoding"]
    BT --> SYN[("Synthetic pairs<br/>noisy source · REAL target")]
    PAR --> MIX["Temperature-sampled mix<br/>q_p proportional to D_p^(1/T)"]
    SYN --> MIX
    MIX --> TRAIN["Train S->T"]
    TRAIN -.->|"round 2 seeds a better<br/>reverse model"| BT

    style PAR fill:#1d3557,color:#fff
    style FILT fill:#9d0208,color:#fff
    style BT fill:#2d6a4f,color:#fff
    style SYN fill:#2d6a4f,color:#fff
    style MIX fill:#bc6c25,color:#fff

The colour key, republished

Every diagram in this repository uses one key, and it is repeated here so this chapter stands alone. It is the key published in The ladder:

Read against that key, the three diagrams in this chapter say: mined parallel corpus is blue because it is the authoritative training data everything else exists to manufacture; the filter bank is red because it discards 60-80% of candidates rather than downweighting them, and a pair thrown away at ingest is not recoverable later; the two back-translation boxes are green because they answer the demand for training pairs without going to the authoritative corpus, which for 9,800 directions does not exist; and the temperature mix is orange because the ratio it enforces is forced by data imbalance, not by compute.

Mining bitext, and why raw cosine fails

The scoring rule that turns a nearest-neighbour search into a usable bitext miner is not the obvious one, and the obvious one fails for a reason you can name.

Every sentence in both crawls is turned into an embedding, and a multilingual encoder is one trained so that a sentence and its translation land in nearly the same place. Index one language’s embeddings, query with the other’s, and take the closest match. Closeness is measured by cosine similarity: the cosine of the angle between two vectors, which is 1.0 when they point the same way and 0 when they are unrelated. Ranking by raw cosine similarity does not work, and the reason is structural.

Some sentences are hubs: "Yes.", "Click here to continue.", "Copyright 2019." carry so little content that they sit near the centroid of the embedding space — the average position of all the vectors — and are therefore near everything. Rank by raw cosine and your top matches are dominated by hubs paired with unrelated hubs, in every language.

The fix is to normalize by local density — the margin criterion:

margin(x, y)  =  cos(x, y)  /  mean over the k nearest neighbours of x and of y of cos(., .)

  a genuine translation pair:  cos = 0.82,  local mean = 0.51  ->  margin 1.61
  a hub paired with a hub:     cos = 0.79,  local mean = 0.77  ->  margin 1.03

The margin asks “is this pair unusually close for these two sentences”, which is the question raw cosine cannot ask. Threshold at roughly 1.06 and you trade recall for precision. Recall is the fraction of the genuine translation pairs in the crawl that you actually keep; precision is the fraction of the pairs you kept that are genuine. Raising the threshold keeps fewer pairs and makes the kept ones cleaner. That is the correct direction here, for the reason derived next.

Filtering: why precision beats recall, in arithmetic

A small misalignment rate is not a small problem, and the number that proves it is what justifies a filter bank that throws away most of what it sees.

A misaligned pair is not a neutral zero-information example. It is an explicit gradient step teaching the model “produce fluent target text that is only loosely related to this source.” That is the hallucination failure mode, taught deliberately.

corpus                 500M pairs
misalignment rate      2%
                       ------------
                       10M training examples whose lesson is "invent content"

Compare that 10M against the supply table above: a mid-resource pair has 100k to 10M real pairs, and a low-resource pair has under 100k. Ten million bad examples is more supervised signal than most low-resource pairs have in total. A 2% misalignment rate is not a 2% quality tax; it is a dedicated hallucination curriculum larger than your entire Icelandic corpus.

That is why the filter bank is precision-first. The code below is the ingest filter: seven checks, each of which is a reject rule returning a code, followed by assertions that fire each rule on its own input. Read keep_pair first and the test harness second — the harness exists to show that each filter catches exactly the thing it claims to catch.

"""Bitext quality filters. Precision-first: every filter is a reject rule."""
import re

MIN_MARGIN = 1.06
LEN_RATIO = (0.5, 2.0)      # target chars / source chars, per-pair calibrated

# Ingest-side numeral check. It is deliberately weaker than the serving-side
# guard in failure mode 3 below: at 500M pairs you can afford one regex per
# side and nothing more, and a false reject at ingest costs one training pair
# while a false reject at serve time costs a user a translation.
DECIMAL_COMMA_LOCALES = {"de", "fr", "es", "it", "pt", "nl", "pl", "ru", "tr", "is"}
NUM = re.compile(r"\d+(?:[.,]\d+)*")     # separators between digit runs only


def numeral_values(text: str, locale: str) -> list:
    out = []
    for m in NUM.finditer(text):
        raw = m.group()
        raw = (raw.replace(".", "").replace(",", ".")
               if locale in DECIMAL_COMMA_LOCALES else raw.replace(",", ""))
        out.append(raw.rstrip(".") or "0")
    return sorted(out)


def keep_pair(src, tgt, pair_cfg, models) -> tuple[bool, str]:
    if models.langid(src) != pair_cfg.src_lang:
        return False, "F01 source language mismatch"
    if models.langid(tgt) != pair_cfg.tgt_lang:
        return False, "F02 target language mismatch"

    ratio = len(tgt) / max(len(src), 1)
    lo, hi = pair_cfg.len_ratio or LEN_RATIO
    if not lo <= ratio <= hi:
        return False, "F03 length ratio outside calibrated band"

    if models.margin(src, tgt) < MIN_MARGIN:
        return False, "F04 alignment margin below threshold"

    # Numerals and entities must survive translation. A pair where they do not
    # is either misaligned or is itself teaching entity mangling.
    if (numeral_values(src, pair_cfg.src_lang)
            != numeral_values(tgt, pair_cfg.tgt_lang)):
        return False, "F05 numeral set differs across sides"

    # Training on the web means training on other systems' output. Left in,
    # it entrenches our own past errors as ground truth.
    if models.mt_detector(tgt) > 0.85:
        return False, "F06 target side is probably machine-translated"

    if models.minhash_seen(src, tgt):
        return False, "F07 near-duplicate of an existing pair"

    return True, ""


# --- Every filter is a REJECT rule, so the test is that each one fires on its
# --- own input and that a clean pair survives all seven.
class _Cfg:
    src_lang, tgt_lang, len_ratio = "en", "de", None


class _Models:
    """Defaults say 'clean'; each case overrides exactly one signal. langid is
    a lookup, so no test depends on a real classifier's opinion."""

    LANG = {"Ship 12 units by Friday.": "en",
            "12 Einheiten bis Freitag liefern.": "de",
            "13 Einheiten bis Freitag liefern.": "de",
            "Bonjour tout le monde.": "fr",
            "Hello world again.": "en",
            "Ja.": "de"}

    def __init__(self, margin=1.40, mt=0.10, dup=False):
        self._margin, self._mt, self._dup = margin, mt, dup

    def langid(self, t):
        return self.LANG[t]

    def margin(self, s, t):
        return self._margin

    def mt_detector(self, t):
        return self._mt

    def minhash_seen(self, s, t):
        return self._dup


SRC = "Ship 12 units by Friday."
TGT = "12 Einheiten bis Freitag liefern."

assert keep_pair(SRC, TGT, _Cfg(), _Models()) == (True, "")
assert keep_pair("Bonjour tout le monde.", TGT, _Cfg(),
                 _Models())[1].startswith("F01")
assert keep_pair(SRC, "Hello world again.", _Cfg(),
                 _Models())[1].startswith("F02")
assert keep_pair(SRC, "Ja.", _Cfg(),
                 _Models())[1].startswith("F03")   # 3 chars against 24 -> 0.125
assert keep_pair(SRC, TGT, _Cfg(),
                 _Models(margin=1.03))[1].startswith("F04")   # hub paired with hub
assert keep_pair(SRC, "13 Einheiten bis Freitag liefern.", _Cfg(),
                 _Models())[1].startswith("F05")
assert keep_pair(SRC, TGT, _Cfg(), _Models(mt=0.91))[1].startswith("F06")
assert keep_pair(SRC, TGT, _Cfg(), _Models(dup=True))[1].startswith("F07")
# And the reason F05 is a reject rather than a downweight: 1.234,5 (de) and
# 1,234.5 (en) are the SAME number, so the filter must not fire on notation.
assert numeral_values("Der Preis ist 1.234,5 EUR.", "de") == \
       numeral_values("The price is 1,234.5 EUR.", "en")
# A separator may join digit runs, never span a list comma or a space. A class
# that spans them reads "1, 23" as the single number 123 and "12, 3" as the
# same one, so a mangled pair survives ingest looking clean.
assert numeral_values("Rooms 1, 23 are open.", "en") == ["1", "23"]
assert numeral_values("Salles 12, 3 sont ouvertes.", "fr") == ["12", "3"]
print("bitext filters: clean pair kept, F01-F07 each fire on their own input")

Every filter in that function is a reject rule, and each returns a code so that a rejection can be counted and audited rather than merely happening. Two of them deserve narration in an interview. F06 uses a classifier that guesses whether a piece of text was produced by a translation system rather than written by a person. It exists because the web now contains an enormous amount of machine-translated text; training on it is a feedback loop that converts your own past errors into ground truth and drags every language toward translationese — prose that is grammatical but reads as though it came out of a translator, with the source language’s word order and idiom showing through. F07 rejects near-duplicates using MinHash, a technique that produces a short fingerprint of a document such that similar documents get similar fingerprints, so you can find near-copies without comparing every pair. It matters because web crawls are pathologically duplicated — boilerplate, syndicated news, mirrored documentation — and duplicated pairs get memorized rather than learned, which is how a model comes to emit a Bible verse when given noise.

Back-translation, derived

The single most valuable data trick in the chapter comes down to one sentence about where the gradient goes.

You have 2M sentence pairs for English-Icelandic and 800M sentences of monolingual Icelandic. The monolingual data is 400x larger and appears useless because it has no source side.

Manufacture one. Train a reverse model that translates Icelandic into English on whatever real data exists, run it over the monolingual Icelandic to produce synthetic English, and then train the forward English-to-Icelandic model on those (synthetic English, real Icelandic) pairs.

Why this works is an asymmetry in where the loss lives. The gradient is the signal that tells each weight in the model which way to move to reduce the loss, and it can only flow back from terms that appear in the loss.

loss  =  - sum over t of  log p( y_t | y_<t, x )
                              ^^^^^^^^^^^^^^^^^
                              gradient flows through the TARGET tokens

The synthetic side is x. Errors in x are input noise — they train the encoder to cope with imperfect input, which acts as a regularizer, meaning something that stops the model latching onto accidental details of the training set and so makes it generalize better. The target side y is real human Icelandic, so the decoder’s implicit language model — the component most starved by a small parallel corpus — is trained on genuine, fluent text of the kind it will actually be asked to produce.

Now reverse it. Forward-translation — translating monolingual English through your own model to get synthetic Icelandic — puts machine output on the target side, where the gradient is. That trains the model to imitate its own output distribution: self-distillation of its own biases, its own translationese, its own systematic errors. It is not merely weaker; it actively entrenches the errors you are trying to remove.

Back-translation puts the noise where the gradient is not. Forward-translation puts the noise exactly where the gradient is. That single sentence is the whole mechanism.

Three practical points that separate people who have run this from people who have read about it:

1. Decode the back-translations with sampling, not beam search. Sampling means drawing each next word from the model’s probability distribution; beam search means searching for the single highest-scoring sentence. Beam output is the mode of the distribution — the most likely sentence — which is clean, low in entropy (a measure of how much variety a distribution contains) and far less varied than real source text. Train on it and the model fits a narrow synthetic-input distribution and then degrades on real input. Sampled or noised-beam output has realistic entropy. This is worth several chrF points and is the most commonly skipped step.
2. Cap the synthetic ratio. Past roughly 3:1 synthetic-to-real, the target distribution starts to be dominated by whatever monolingual domain you crawled, and the model drifts.
3. Iterate, but stop at three rounds. A better forward model gives a better reverse model, which gives better back-translations. Round 2 is worth about a third of round 1; round 3 about a third of round 2.

The table below measures the effect on an English-to-Icelandic development set. Three columns, glossed once and used throughout the chapter: chrF (character n-gram F-score) counts how many short character sequences a candidate shares with the reference and is derived in the metric ladder; COMET (crosslingual optimized metric for evaluation of translation) is a learned scorer, a model trained on human quality judgements that predicts what a human would have said, on a 0-to-1 scale; hallucination rate is the fraction of outputs that assert something the source did not say.

Row 2 is the row to point at: forward-translation raised a surface metric slightly and made hallucination worse, which is exactly what the asymmetry argument predicts.

The data mix: temperature sampling, with the number

How often should each language pair appear in a training batch, when one pair has a thousand times more data than another? The answer is a single exponent.

A batch is the group of examples the model sees together in one training step. If batches are drawn in proportion to corpus size, a small pair essentially never appears.

The fix is to flatten the distribution: sample pair p in proportion to D_p^(1/T), where D_p is its corpus size and T is a temperature knob. Raising a number to the power 1/T compresses differences — the bigger T is, the more it compresses. T = 1 is the exponent 1, which leaves the natural proportions alone. T heading to infinity drives the exponent to 0, and anything to the power 0 is 1, which is a uniform mixture.

Work it on the two extremes of the actual corpus:

en-fr:  2,000,000,000 pairs
en-is:      2,000,000 pairs        ratio 2e9 / 2e6  =  1000 : 1

T = 1  (natural)     exponent 1     ratio stays 1000 : 1
                                    ->  en-is is 0.1% of batches; never learned
T = inf (uniform)    exponent 0     ratio 1 : 1
                                    ->  en-is repeats ~1000 epochs; memorized
T = 5                exponent 1/5   ratio = 1000^(1/5)
                                          = (10^3)^(1/5)
                                          = 10^(3/5)
                                          = 10^0.6
                                          = 3.98 : 1

A fifth root turns three orders of magnitude into a factor of four. That is the entire trick, and 3.98 is the number to have at hand.

The cost is the mirror image, and it is worth doing the division. An epoch is one full pass over the training data. English-French has 1000x the sentences but is only sampled 3.98x as often, so each individual English-Icelandic sentence gets shown

1000 / 3.98  =  251 times more often per epoch

than each individual English-French sentence. At roughly 250 repetitions a sentence is at real risk of being memorized rather than learned. That is why F07 (near-duplicate rejection) and a per-pair cap on how many times a pair may be repeated are not optional hygiene — they are what makes the temperature safe.

Assumptions in this stage.

• State out loud: the corpus sizes driving the arithmetic — 2M real English-Icelandic pairs against 2B English-French, 800M monolingual Icelandic sentences, a 2% misalignment rate in raw mined bitext, and T = 5. Different numbers move the fifth-root result and the 251x repetition figure, and nothing else.
• Ask, never assume: whether you are permitted to train on crawled web text at all, and in which jurisdictions. That is a legal answer, not a technical one. A “no” deletes the mining half of this section outright: what survives is supervised training on licensed corpora for the roughly 100 directions that have them, plus back-translation from licensed monolingual text, and coverage falls from 9,900 directions to a few hundred.
• Load-bearing: that back-translation’s noise really does land on the input side and not the target side. The whole low-resource strategy, and the 41.2-to-49.3 chrF jump in the table above, rest on that one asymmetry. If machine output on the source side degraded the model the way it does on the target side, there would be no affordable way to serve 9,800 directions and the honest design would be to support far fewer languages properly.
• Also load-bearing: that the mined corpus can be filtered to well under 2% misalignment. At 2% you ship ten million examples that teach the model to invent content — more supervised signal than most low-resource pairs have in total. A design that cannot hit that filter precision has a hallucination rate set by its data pipeline, and no serving-time guard can fix it.

Model choice: one multilingual model, and why

The obvious architecture — one specialist model per language pair — dies on a serving-cost argument rather than a quality argument, and the single shared model that replaces it gives something up in exchange.

The pairwise fleet dies on residency, not on training

Price the specialist-per-pair fleet and it fails on a cost that traffic never pays down.

The instinct is N × (N-1) specialist models, one per direction. Most candidates then argue about training cost. Price the serving side instead, because that is where it actually fails.

The key idea is weight residency: a model can only answer a request if its weights are sitting in GPU memory at that moment, and you rent that memory by the hour whether or not a request arrives.

Weights are stored in fp16, a 16-bit floating-point number format, so each parameter costs 2 bytes. Each specialist is 0.4B parameters:

9,900 directed pairs × 0.4B params × 2 bytes (fp16)  =  7,920 GB of weights
GPU usable memory                                     =  70 GB
                                                        ---------------------
7,920 / 70                                            =  114 GPUs to hold the
                                                         weights, before serving
                                                         one single request

Now overlay the traffic on those idle weights. Average traffic is 13,889 QPS (derived in scale and cost). The top 50 pairs take 80% of it, so the remaining 9,850 pairs share the other 20%:

13,889 × 0.20                =  2,778 QPS for the whole tail
2,778 QPS  /  9,850 pairs    =  0.28 QPS per tail pair

one GPU can serve 620 QPS of this model, and it is serving 0.28
0.28 / 620                   =  0.00045  =  0.045% utilization

0.045% utilization is the number to say out loud. You are renting a chip and using one part in 2,200 of it.

The obvious escape is to load a model only when a request for it arrives, and it does not work either. Each model is 0.8 GB (0.4B params × 2 bytes). Reading that from NVMe — the fast solid-state storage attached to the server — at 3 GB/s takes

0.8 GB / 3 GB/s  =  0.267 s  =  267 ms, call it 270 ms

of cold-start latency, meaning the delay before a model that was not already resident can answer at all, added to the first request of every rare pair. And for a rare pair at 0.28 QPS, requests almost never arrive back to back, so “the first request” is essentially every request. That 270 ms alone nearly consumes the 300 ms p95 budget before the model has produced a single token.

A pairwise fleet’s cost is set by weight residency, and residency does not amortize over traffic. One multilingual model is O(1) in the number of pairs — meaning its cost does not grow as pairs are added. One set of weights serves all 9,900 directions, and every GPU can serve every request, so the scheduler that groups concurrent requests into a batch can pack freely instead of only combining requests that happen to want the same pair.

Zero-shot transfer, and its specific failure

How does a model translate a direction it was never trained on — and what is the specific way that capability breaks?

A model trained only on English-to-everything and everything-to-English can translate Icelandic into Swahili without ever having seen that pair — the zero-shot case. The mechanism is that the encoder maps source text into a representation that is largely language-agnostic, so Icelandic and English sentences with the same meaning land in nearly the same place, and the target-language tag then tells the decoder which language to write in. Quality sits well below a supervised model and well above nothing — and nothing is the honest alternative for 9,800 directions.

The characteristic failure is off-target translation: asked for Icelandic to Swahili, the model emits English, because English was the dominant target during training and the tag is a single token competing against that overwhelming prior. Rates of 20-50% on some directions are normal for a naively trained model. Four fixes, cheapest first. Put the target-language tag as the first token the decoder emits rather than as a prefix on the source, so that the decoder cannot learn to attend away from it — free. Add a wrong-language penalty from a cheap language-ID classifier inside the beam search, which costs about 2% of decode time. Reject and re-decode at serve time when a classifier says the output language disagrees with what was requested, which costs one extra decode on 1-3% of tail traffic. And add non-English-centric synthetic data by back-translating one non-English language into another through the model itself, which creates the missing pair at the cost of compounding the model’s own errors.

The curse of multilinguality, in parameters

“Adding languages hurts” is a slogan until you do the division — and the division is what lets you answer the follow-up question about adding thirty more.

Adding languages to a fixed parameter budget helps low-resource languages through transfer — a language borrowing structure from its relatives in the same model — up to a point, and then hurts everything. This is called the curse of multilinguality, and division makes the shape obvious.

The idea is that the model’s parameters are a fixed pot. Some of them are shared across all languages, and some are effectively spent on one language’s own quirks. Add languages and that second pot gets divided more ways.

model                      400M params
shared subword embedding   262M           (derived in the tokenizer section)
transformer body           400M - 262M  =  138M
say half the body is effectively language-private:   138M / 2  =  69M

  L = 25 languages   ->  69M / 25  = 2.76M private params per language
  L = 100 languages  ->  69M / 100 = 0.69M private params per language
                         --------------------------------------------
                         2.76 / 0.69 = 4x cut in per-language capacity

The “half the body is language-private” split is a modelling convenience, not a measurement. Move it to a third or two-thirds and both numbers move together; the 4x ratio between them does not move at all, because it is just 100 / 25.

The 262M figure on the second line is derived in the vocabulary is most of the model; it is the size of the lookup table that turns tokens into vectors, and it is not available for learning anything language-specific.

Transfer partially offsets it: a language with close relatives in the set borrows their structure and can come out ahead. A language isolate — one with no known relatives, such as Basque — borrows nothing and takes the full 4x loss.

So “should we add 30 more languages?” has exactly one honest answer: yes, if you also scale parameters; otherwise you are paying for them with the low-resource languages you already have. The number of languages L* at which adding one more starts to hurt moves higher as the model grows, which is why this is a capacity question and not a linguistics question.

There are two ways to buy per-language capacity without buying more arithmetic per token, where the arithmetic is counted in FLOPs, floating-point operations:

• Mixture of experts (MoE). The model holds many parallel copies of some layers — the experts — and a small router sends each token to only one or two of them. Total parameters grow while the parameters used per token stay fixed, so per-language private capacity grows at constant per-token compute. The cost is memory residency: every expert has to be sitting in GPU memory even though most are idle for any given token, which is precisely the constraint that killed the pairwise fleet. So MoE only pays where you can afford the memory. Hence the split used throughout this chapter: MoE in the big lane, a dense distilled model in the throughput lane.
• Language-specific adapters. Small per-language weight patches bolted onto a shared trunk. Attractive on paper, and rejected here for a serving reason given in alternatives considered — a batch can only contain requests that share weights.

Assumptions in this stage.

• State out loud: a 400M-parameter throughput model and an 8B mixture-of-experts model in the big lane, with roughly half of the transformer body treated as effectively language-private. That last figure is a modelling convenience, not a measurement, and moving it moves both sides of the 2.76M-versus-0.69M comparison without changing its direction.
• Ask: how many languages the product intends to support in two years. The honest answer to “add 30 more” depends entirely on whether the parameter budget grows with them.
• Load-bearing: that one shared model can serve all 9,900 directions at acceptable quality via zero-shot transfer. If zero-shot quality on non-English-centric pairs were unusable rather than merely worse, no amount of routing or filtering saves this design. The alternative is a much smaller supported language set with real data behind each pair, which is a different product.
• Also load-bearing: that there is enough GPU memory to hold one 8B mixture-of-experts model resident. The MoE lane’s entire justification is that residency is affordable for one model and not for 9,900. If it were not affordable for one either, the big lane becomes a dense model and the low-resource quality argument weakens with it.

Tokenization: the decision that sets cost and quality for your worst languages

Most candidates skip tokenization entirely, yet it is where the fairness, cost and latency of your hundred languages are jointly decided — by a choice someone made before training started.

The design is one shared subword vocabulary across all 100 languages. A subword vocabulary is a fixed list of text fragments — whole common words, prefixes, suffixes, single characters — and tokenizing means greedily covering an input with pieces from that list. It is fit with a unigram language model, an algorithm that searches for the set of pieces that explains a training corpus with the fewest pieces on average, run over a temperature-sampled corpus so that no single language dominates. See Tokens for the full mechanism; what follows is what changes at 100 languages.

Why shared, not per-language:

1. Transfer flows through shared tokens. Cognates, shared scripts, digits, punctuation, and named entities occupy the same embedding rows across languages. With disjoint vocabularies, Information in English and Information in German would be two unrelated rows and nothing would transfer between them.
2. Zero-shot depends on it. The shared representation the encoder builds is anchored by tokens that appear in more than one language. Remove the anchors and the directions you never trained on have nothing to stand on.
3. One embedding matrix instead of 100. Which is also the problem.

The vocabulary is most of the model

Multiply two numbers most people never multiply, and vocabulary size turns into a serving-memory decision.

The first thing a model does with a token is look it up in an embedding matrix: one row of numbers per vocabulary entry, d_model numbers wide, where d_model is the width of every internal representation in the model. So the table has as many rows as the vocabulary has entries and as many columns as the model is wide, and its parameter count is just the product.

A model normally has two such tables — one to turn input tokens into vectors, and one to turn the final vector back into a score for every vocabulary entry. Tying them means using the same matrix for both, which halves the count.

Multiply the two numbers:

vocab 256,000  ×  d_model 1,024   =  262,144,000 params  =  262M

untied (separate input and output matrices):  2 × 262M  =  524M params
                                              -> larger than the entire 400M model
tied (one matrix used twice):                 262M of a 400M model
                                              262 / 400  =  66%

Two-thirds of a 400M multilingual model is a lookup table. That is the number most people never compute.

In a small multilingual model the vocabulary is not a preprocessing detail; it is the majority of the parameters. Three consequences you should volunteer:

1. Tie the input and output embeddings. At this size that is mandatory, not optional — untied costs you 524M parameters for a 400M model, which is incoherent.
2. Treat vocabulary size as a serving-memory decision, reviewed with the same seriousness as the number of layers.
3. Consider a factorized embedding. Instead of one wide table, keep a narrow one and multiply it back up:

one wide table:   256,000 × 1,024                 =  262.1M params

factorized:       256,000 × 128     =  32.77M
                      128 × 1,024   =   0.13M
                                       -------
                                        32.90M params

262.1 / 32.90  =  8x fewer parameters

The cost is that every token’s identity now has to squeeze through 128 numbers instead of 1,024 — a bottleneck worth about 0.8 chrF on high-resource pairs.

Script imbalance, and the compounding it causes

A single tokenizer decision sets cost, latency and quality for a language simultaneously, all in the same direction — which is why it is a fairness question rather than a preprocessing question.

The vocabulary is fit on a corpus, and the algorithm spends its fixed budget of pieces wherever the text is. If that corpus is 40% English, the budget goes on English word fragments. A language at 0.05% of the corpus gets almost no dedicated pieces and falls back toward individual characters or raw bytes.

The measurement that exposes this is fertility: the average number of tokens the tokenizer produces per word of a language. Fertility 1.0 means one token per word; fertility 5.2 means the model has to emit five pieces to write one word.

In the table below, read down the corpus-share column and then down the fertility column — they move in opposite directions, and that is the whole finding. The last column is just fertility divided by English’s 1.15.

Two of those script labels are worth a word. Turkish is agglutinative, meaning it builds long words by gluing suffixes onto a stem, so a whole English clause can be one Turkish word and a tokenizer without dedicated pieces for those suffixes shatters it. Ge’ez is the script Amharic is written in, and its several hundred distinct characters are essentially absent from a corpus that is 40% English.

Three consequences follow, and the point is that they compound rather than trade off. Each one is worked out below on Amharic’s 4.5x.

1. Cost. The same sentence costs 4.5x more tokens in Amharic. Your user-facing price is per character, so the user never sees this; your GPU bill sees nothing else.

2. Latency. Decode is one sequential step per token (The kv cache the most important mechanism in this chapter), so 4.5x the tokens is 4.5x the decode — and only the decode. Everything else in the request is fixed cost. Take the Lane B line items from the latency budget below:

Lane B total                    140 ms
  of which decode                72 ms   <- scales with token count
  everything else                68 ms   <- network, normalize, queue,
                                            prefill, guards: does not move

Amharic:   68 + (4.5 × 72)  =  68 + 324  =  392 ms
                392 / 140   =  2.8x the English request

392 ms against a 300 ms p95 target. The budget is blown by a tokenizer decision, on a request the model handles no differently.

3. Quality. A 60-word sentence becomes 60 × 5.2 = 312 tokens in Amharic against 60 × 1.15 = 69 in English. The cost of attention — the mechanism by which every token in a sentence looks at every other token — grows with the square of the sequence length (Attention and why context costs what it does):

(312 / 69)^2  =  4.52^2  =  20x the attention compute of the English equivalent

And on top of the compute, the model must compose meaning out of character-level fragments it has seen in far fewer contexts than it has seen -ing.

The languages that get the worst quality also get the worst latency and the worst cost, and all three come from one decision made when someone fit the tokenizer. That is the sentence to say out loud, because it reframes tokenization from a preprocessing step into a fairness and capacity decision.

Three fixes follow from the mechanism. Temperature-sample the vocabulary-fitting corpus with the same T you use for training, which most teams forget — they sample the training mix and fit the vocabulary on the raw one. Floor each language’s allocation at some minimum number of pieces, so no script can be squeezed out entirely. And track fertility per language as a first-class dashboard metric, so that a vocabulary change cannot silently regress Amharic while improving the average.

Assumptions in this stage.

• State out loud: a 256,000-entry shared vocabulary at d_model = 1,024, and the corpus shares in the fertility table. Different sizes move the 262M figure and the 66% share proportionally; the argument that the embedding table dominates a small model survives any plausible choice.
• Ask: which languages the product actually promises to serve well. The vocabulary budget is finite, and flooring every one of a hundred languages costs pieces that the top fifty currently have.
• Load-bearing: that the tokenizer is fixed before training and cannot be changed afterwards without retraining from scratch. That is what makes this an architecture decision rather than a tuning knob, and it is why fertility has to be checked before the training run rather than diagnosed after it. If vocabularies could be swapped cheaply on a trained model, this whole section shrinks to a maintenance task.
• Also load-bearing: the 300 ms p95 budget, again. The 392 ms Amharic figure is only a failure because of it.

Training

Five training stages run in order, and one wrinkle among them is what an interviewer will push on.

The table is a pipeline: each stage takes the model the previous stage produced. Stage 1 is where almost all the compute goes; stages 3 to 5 are cheap and are where most of the shipped quality is decided.

Three stage names need unpacking. Distillation (stage 4) trains a small “student” model to imitate a large “teacher” model, so that the student inherits behaviour it was too small to learn on its own. Quantization (stage 5) stores the weights in a smaller number format — here int8, 8-bit integers instead of 16-bit floats — which halves the bytes that must be read per decode step and therefore roughly doubles decode speed, at some cost in accuracy. A held-out set is data deliberately kept out of training so that measuring on it is not measuring memorization.

Stage 4 has a wrinkle worth naming. Sequence-level distillation means training the student on the whole sentence the teacher actually produced, rather than on the teacher’s logits — the raw scores it assigned to every possible next token before those scores were turned into probabilities. It is the back-translation asymmetry read in reverse: here the machine output lands on the target side, where the gradient is, so the student learns the teacher’s mode. That mode is cleaner and less varied than real data, and therefore easier for 0.4B parameters to fit than the real data’s full diversity. The student beats a 0.4B model trained directly on real data. It also inherits the teacher’s hallucinations, so stage 5’s gate must test for those and not just chrF.

Assumptions in this stage.

• State out loud: an 8B teacher distilled into a 0.4B student, 500M-2B filtered pairs for pretraining, 1-5M human-verified pairs for the clean fine-tune, and two to three back-translation rounds. Those set the training bill, not the design.
• Ask: whether 1-5M human-verified pairs actually exist for your languages, or whether “clean in-domain data” means a few thousand for most of them. The answer decides whether stage 3 is a real stage or a rounding error outside the top fifteen pairs.
• Load-bearing: that sequence-level distillation preserves enough quality for the student to carry 60% of traffic. The entire two-lane cost structure depends on the small model being close enough to the big one on high-resource pairs. If it is not, everything routes to the big lane: 1,585 GPUs at $76,080/day, which is the +$62,736 row of the lever table — a change, not a total — added to the $13,344 decode baseline.
• Also load-bearing: that the student’s inherited hallucinations are detectable by the stage-5 gate. If they are not, you have shipped the teacher’s worst behaviour on 60% of traffic with no measurement of it.

Offline metrics

The launch decision runs on a ladder of measurements, and building it starts with dismantling the metric everyone quotes — the part interviewers push hardest on.

BLEU, and exactly what it cannot see

On one worked pair of sentences, BLEU ranks a factual error above a correct paraphrase — and the reason is arithmetic about window positions, not anything to do with meaning.

BLEU (bilingual evaluation understudy) compares a candidate translation against a human reference by counting shared n-grams, an n-gram being a run of n consecutive tokens. It is the geometric mean of four precisions — one each for runs of length 1, 2, 3 and 4 — with a length correction:

BLEU  =  BP · exp( sum over n=1..4 of (1/4) · log p_n )

p_n   =  clipped n-gram precision
      =  (candidate n-grams appearing in the reference, clipped to the reference count)
         / (total candidate n-grams)

BP    =  1                    if c > r
      =  exp(1 - r/c)         otherwise        c = candidate length, r = reference length

Read the pieces one at a time.

• p_n is a clipped precision: of all the n-grams in the candidate, what fraction also appear in the reference. “Clipped” means a candidate cannot get credit for repeating an n-gram more times than the reference contains it — otherwise the the the the would score well against a reference containing one the.
• exp( sum of (1/4)·log p_n ) is just a geometric mean written the way it is implemented: multiply the four precisions together and take the fourth root. This matters enormously, because a geometric mean containing a single zero is zero, no matter how good the other three factors are.
• BP is the brevity penalty. It scales the score down when the candidate is shorter than the reference, and leaves it alone (BP = 1) when the candidate is at least as long.

The brevity penalty exists only because precision alone is maximized by emitting a single high-confidence word and stopping. That patch tells you what BLEU fundamentally is: a string-overlap statistic with a length correction bolted on.

The demonstration, counted by hand

One reference and two candidates. Each candidate differs from the reference by exactly one word.

REFERENCE     The meeting was postponed until Thursday .
CANDIDATE A   The meeting was delayed   until Thursday .    meaning preserved
CANDIDATE B   The meeting was postponed until Tuesday  .    meaning destroyed

Both candidates are 7 tokens long, same as the reference, so c = r and BP = 1 for both. All the action is in the four precisions.

Count them for Candidate A (delayed replaces postponed at position 4). List the windows and mark which survive:

1-grams (7)  The ✓ meeting ✓ was ✓ delayed ✗ until ✓ Thursday ✓ . ✓        6/7
2-grams (6)  [The meeting]✓ [meeting was]✓ [was delayed]✗
             [delayed until]✗ [until Thursday]✓ [Thursday .]✓             4/6
3-grams (5)  [The meeting was]✓ [meeting was delayed]✗ [was delayed until]✗
             [delayed until Thursday]✗ [until Thursday .]✓                2/5
4-grams (4)  [The meeting was delayed]✗ [meeting was delayed until]✗
             [was delayed until Thursday]✗ [delayed until Thursday .]✗    0/4

Every 4-gram window contains position 4, so p_4 = 0. Now the geometric mean:

BLEU-A  =  ( 6/7 × 4/6 × 2/5 × 0/4 ) ^ (1/4)
        =  ( 0.857 × 0.667 × 0.400 × 0.000 ) ^ (1/4)
        =  0 ^ (1/4)
        =  0.000

Now Candidate B (Tuesday replaces Thursday at position 6):

1-grams (7)  The ✓ meeting ✓ was ✓ postponed ✓ until ✓ Tuesday ✗ . ✓       6/7
2-grams (6)  [The meeting]✓ [meeting was]✓ [was postponed]✓
             [postponed until]✓ [until Tuesday]✗ [Tuesday .]✗             4/6
3-grams (5)  [The meeting was]✓ [meeting was postponed]✓
             [was postponed until]✓ [postponed until Tuesday]✗
             [until Tuesday .]✗                                           3/5
4-grams (4)  [The meeting was postponed]✓ [meeting was postponed until]✓
             [was postponed until Tuesday]✗ [postponed until Tuesday .]✗  2/4

Two 4-gram windows survive because they sit entirely to the left of position 6. So:

BLEU-B  =  ( 6/7 × 4/6 × 3/5 × 2/4 ) ^ (1/4)
        =  ( 0.857143 × 0.666667 × 0.600000 × 0.500000 ) ^ (1/4)
        =  ( 0.171429 ) ^ (1/4)
        =  0.6435

Side by side:

BLEU ranks the factual inversion above the correct paraphrase, and the margin is the whole scale.

The mechanism is positional, not semantic. delayed sits at token 4 of 7 — the exact centre, the one position in a 7-token sentence that every 4-gram window covers. Tuesday sits at token 6, near the end, where only two 4-grams reach it and The meeting was postponed and meeting was postponed until both survive intact. A substitution’s BLEU cost is set by where in the sentence it lands, and meaning has no representation in the statistic at all.

The same error at all seven positions

To prove that it is position and not meaning, replace one token with a wrong word at each of the seven positions in turn. Nothing else changes: same length, same single error, same semantic damage. The four precisions and the resulting BLEU:

Note that p_1 is 6/7 in every row — the unigram precision cannot tell these seven cases apart at all. Everything that varies is the higher-order columns, and they vary only with position.

Why the “4-grams broken” column controls the score: a token at position i sits inside as many 4-gram windows as there are windows covering it — one at each end of the sentence, four in the centre. Break all four and p_4 = 0, and since BLEU is a geometric mean over p_1 through p_4, one zero factor takes the whole product to zero no matter how good the unigrams are. In a 7-token sentence, position 4 is the only place a single substitution can do that. The curve is a triangle centred on the sentence, and it is a property of window arithmetic, not of words.

That is a stronger indictment than “BLEU cannot tell them apart.” It can tell them apart, and it prefers the wrong one — a system tuned to maximize BLEU is being paid to move its errors toward the ends of sentences.

Candidate A’s 0/4 exposes a second problem: the geometric mean makes sentence-level BLEU zero whenever any order has no match, which is most single sentences. BLEU is a corpus-level statistic that people routinely misuse at the sentence level with an arbitrary smoothing hack — and here the smoothing scheme you pick decides whether the correct translation scores zero or something.

And two comparability failures that get stated wrongly in interviews all the time:

Not comparable across languages. p_n is computed over tokens of the target language. Morphologically rich languages pack meaning into fewer, longer words, so one wrong inflection destroys up to four n-grams at once and the achievable ceiling is structurally lower. Chinese and Japanese have no whitespace, so the score is a function of whichever segmenter you ran. A BLEU of 32 on en->de and 32 on en->fi are not the same quality, and a paper claiming “our Finnish score is lower because Finnish is hard” is describing the metric, not the system.

Not comparable across tokenizations. Identical model output, different detokenization — the step that reassembles tokens back into readable text — gives a different score, because punctuation splitting, casing and Unicode normalization all move it by several points. This is what sacreBLEU exists for: a standard implementation that fixes the tokenization and emits a signature string recording exactly which settings produced the number. A BLEU number without a sacreBLEU signature is not a measurement, it is a claim.

Not sensitive in the range that matters. Modern systems cluster tightly, and a 0.3 BLEU difference is inside the noise of which reference translator you hired. Any reported difference needs paired bootstrap resampling: repeatedly draw a random test set of the same size by sampling your sentences with replacement, score both systems on each draw, and see how often the winner changes. Most published differences do not survive it.

The metric ladder

BLEU dismantled, what remains is to arrange the measurements you can afford by how often you can run them — because the trustworthy ones are slow and the fast ones are shallow.

One word first, because it is used constantly from here on and never defined. An eval is a fixed measurement suite — a frozen test set, a scoring rule and a threshold — run against a candidate model on a schedule. It is not evaluation in general; the point of the word is that the set does not move, so two runs are comparable.

The ladder has three tiers, and the tier labels are the whole point: they say how often you can afford to run each one.

The cheap tier is marked per commit · seconds · deterministic, and it holds chrF2, described as chrF2 character n-gram F-score beta=2, recall-weighted — a character-level overlap score that leans toward recall and is defined properly below.

The middle tier is marked per release · minutes · GPU, and it holds the learned metrics: COMET / BLEURT learned regressor on human judgements — BLEURT being bilingual evaluation understudy with representations from transformers, which despite the name is a learned regressor rather than an n-gram count — plus COMET-QE reference-free DEPLOYABLE at serve time, the only one of them usable in production, because production has no reference translation to compare against.

The slow tier is marked pre-launch · days · money: MQM human eval error spans + severities, where professional annotators mark error spans and assign severities.

Cheap feeds middle feeds slow as the gates get stricter, the middle tier also branches to the deployable reference-free variant, and the dotted arrow back from MQM is labelled retrains / recalibrates, because human judgements are the data the learned metrics are fit on.

flowchart LR
    subgraph CHEAP["Per commit · seconds · deterministic"]
        CHRF["chrF2<br/>character n-gram F-score<br/>beta=2, recall-weighted"]
    end
    subgraph MID["Per release · minutes · GPU"]
        COMET["COMET / BLEURT<br/>learned regressor on<br/>human judgements"]
        QE["COMET-QE<br/>reference-free<br/>DEPLOYABLE at serve time"]
    end
    subgraph SLOW["Pre-launch · days · money"]
        MQM["MQM human eval<br/>error spans + severities"]
    end
    CHRF --> COMET --> MQM
    COMET --> QE
    MQM -.->|"retrains / recalibrates"| COMET

    style CHRF fill:#2d6a4f,color:#fff
    style QE fill:#bc6c25,color:#fff
    style MQM fill:#1d3557,color:#fff

Against the colour key: MQM is blue because human judgement is the authoritative copy of “quality” and every other box on the ladder is a cheaper approximation of it; chrF2 is green because it answers the quality question per commit without asking that authoritative copy; COMET-QE is orange because what forces it into the design is not its cost but the absence of a reference at serve time.

chrF2 is the F-score over character n-grams up to length 6. An F-score is the harmonic mean of precision and recall; beta = 2 weights recall twice as heavily as precision. Averaging over n = 1..6 and then combining, rather than taking a geometric mean, is why chrF has no zero cliff: one bad order cannot annihilate the score.

Why it beats BLEU as a default: a wrong Finnish inflection loses a suffix’s characters rather than an entire word plus four n-grams, so it degrades gracefully with morphology — and it needs no word tokenizer at all, which removes BLEU’s whole comparability-across-tokenizations problem.

Be honest about its limit. Run chrF2 on the same Thursday/Tuesday example, with the standard settings (character n-grams 1 through 6, beta = 2, whitespace stripped before counting):

chrF2 compresses the spread that BLEU blew up — 0.72 against 0.84 instead of 0.00 against 0.64 — but it still ranks the factual inversion higher.

The reason is a different one from BLEU’s, and being precise about that is what separates a candidate who has read about these metrics from one who has run them. chrF has no positional term at all: it counts character n-grams as a bag. Thursday and Tuesday share T, u, s, day and the T...day frame; postponed and delayed share almost nothing but ed. The ranking is character overlap, full stop.

Two controls separate the two mechanisms, and they are the pair to quote:

If chrF were positional, the late substitution would score higher — under BLEU’s window arithmetic, position 6 is far cheaper than position 2. It scores lower. So chrF is ranking on character overlap and ignoring position entirely.

So the two metrics agree on the wrong answer here by coincidence, not by a shared flaw — which means fixing one tells you nothing about the other. chrF fixes the morphology problem, not the meaning problem — no surface-overlap statistic can, because the information is not in the surface.

COMET and BLEURT are learned metrics. A multilingual encoder reads the source, the model’s output — the hypothesis — and the human reference, and a small regression head on top predicts the score a human would have given. They agree with human sentence-level judgement at roughly 0.4-0.6 Kendall tau — a rank-correlation measure running from -1 for perfectly reversed to +1 for perfectly matched ordering — against roughly 0.2 for BLEU, and they do separate Thursday from Tuesday. Four caveats to volunteer, because a candidate who presents COMET as the answer gets pushed on all four:

• Scores are only comparable within a checkpoint. A checkpoint is one saved version of the metric model’s weights, and different versions produce different numbers on identical output. Pin the version and put it in the report. A COMET number without a model version is the sacreBLEU-signature problem again.
• Optimizing against it is reward hacking — the general failure where a system maximizes the measurement rather than the thing the measurement stands for. Minimum-Bayes-risk (MBR) decoding generates many candidate translations and keeps the one a metric likes best; run against COMET, it produces outputs that score high and read strangely. Keep the metric you decode against separate from the metric you report.
• It inherits the biases of its training annotations. Whatever the human raters systematically preferred, the metric now systematically prefers.
• The reference-free variant (COMET-QE, where QE stands for quality estimation) is the deployable one, because production has no reference translation to compare against — that is the whole point of translating. It is weaker than the reference-based variant, and it is the single most useful model in the serving path, where it appears as the third layer of the hallucination guard.

MQM — multidimensional quality metrics — is what quality actually means. Professional bilingual annotators mark error spans with categories (accuracy: mistranslation, omission, addition; fluency: grammar, register, terminology) and severities. The score is a negative weighted count of errors per 100 words. It beats a 5-point adequacy scale because it produces diagnostics: “our regression is 60% addition errors” is actionable, “quality fell 0.3” is not.

Price it, because the price is why it sits at the bottom of the ladder. Each line below feeds the next:

annotator            $40/hour, ~60 segments/hour with full MQM

eval round           500 segments × 12 pairs × 3 annotators     =  18,000 judgements
hours of work        18,000 judgements / 60 per hour            =     300 annotator-hours
money                300 h × $40/h                              =  $12,000
wall-clock           300 h / (10 annotators × 8 h/day)          =    3.75 days

Note that the “3 annotators” is redundancy per segment, not headcount: the panel that turns 300 hours into four days is ten people. Halve the panel and the same money takes eight days.

Monthly, launch-blocking, on a fixed set. Not per commit.

The gate that actually protects you

One rule stops a launch from destroying a language nobody on the team speaks.

Per-pair regression gates, not an aggregate. At 100 languages, a pair carrying 0.05% of traffic can go from good to garbage and move the aggregate chrF by 0.003 — well inside the noise of any measurement you would trust. The gate is therefore stated per direction: no directed pair may drop more than 1.0 chrF or 0.02 COMET against the current production model, measured on a fixed 500-segment set for that pair. An aggregate improvement does not buy an exemption.

Assumptions in this stage.

• State out loud: a fixed 500-segment evaluation set per directed pair, thresholds of 1.0 chrF and 0.02 COMET, and MQM at $40 per annotator-hour and 60 segments an hour. Those set the cost and the sensitivity of the gate; different values move both without changing the structure.
• Ask: whether bilingual annotators are procurable at all for your tail languages. For many directed pairs there is no market of people fluent in both sides. Where there is not, MQM is not merely expensive — it is unavailable, and the gate falls back to chrF plus the serving quality-estimation model.
• Load-bearing: that a 500-segment fixed set per pair is representative of live traffic for that pair. If it is not — drawn from news text while the traffic is chat — the gate passes systems that are broken on the real distribution. That is worse than no gate, because it is believed.
• Also load-bearing: that the reference-free quality-estimation model is usable across all 100 languages. Every serving guard and the escalation logic in which lever, in dollars depend on it. On languages where its scores are uninformative, the design has no runtime quality signal at all and falls back to the deterministic checks alone.

Online metrics and A/B

Live traffic offers a handful of behavioural signals — and, at 100 languages, a derivation showing that the launch decision for the languages you most need to protect cannot be an online experiment at all.

The power calculation, and why it argues against a global A/B

Size the experiment, and the size itself proves the tail cannot be launched online.

An A/B test splits live traffic between the current system and the new one and compares a metric. Sizing one needs three inputs:

• Alpha is the false-positive rate you accept. 0.05 means a 1-in-20 chance of calling a win that is not real.
• Power is the chance of detecting a real effect when one exists. 0.8 is standard.
• Effect size is the smallest change you want to be able to see.

Those first two get folded into a single constant. The standard sample-size formula for comparing two proportions is n ≈ 2·(z_alpha + z_beta)^2 · p(1-p) / delta^2, and at alpha 0.05 two-sided and power 0.8 the z values are 1.96 and 0.84, so

2 × (1.96 + 0.84)^2  =  2 × 7.84  =  15.7,  round to 16

That is where the 16 comes from. It is not magic; it is alpha 0.05 and power 0.8 baked into one number.

Now put in real values. Say you want to detect a 0.5% relative change in the copy rate, against a baseline copy rate of p = 8%:

delta      =  0.005 × 0.08          =  0.0004 absolute
             (0.5% OF 8%, not 0.5 percentage points — this is the step
              people get wrong, and it is a factor of 12.5)

n per arm  ≈  16 · p(1-p) / delta^2
           =  16 · 0.08 · 0.92 / (0.0004)^2
           =  1.1776 / 1.6e-7
           =  7,360,000  =  7.36M requests per arm

How long does 7.36M take to collect? At 1.2B requests/day, a 1% experiment allocation gives

1.2e9 × 0.01  =  12M requests/day, split two ways  =  6M per arm per day
7.36M / 6M    =  1.2 days for the global number

1.2 days sounds fine, and it is the trap.

The trap is that the effect is not uniform across pairs. It is concentrated in a handful of them. So you have to slice by pair — and each slice needs its own 7.36M, which a pair at 0.05% of traffic will never accumulate:

tail pair volume in the experiment
  =  1.2e9 requests/day × 0.0005 (share) × 0.01 (allocation)
  =  6,000 requests/day

7.36M / 6,000  =  1,227 days  =  3.4 years

You would need 3.4 years to power an A/B on a tail pair. That is not a tuning problem; it is a structural fact about the traffic distribution.

So the launch decision cannot be an online A/B for the languages you most need to protect. It is:

• Offline per-pair regression gates as the blocker (the gate that actually protects you). This is the real gate.
• Side-by-side preference panels for the tail: show two outputs to bilingual raters, record preference. Vastly more power per observation than a behavioural metric, and it works at 200 segments instead of 7.36M requests. Its limit is that it measures preference, not correctness.
• Online A/B for the high-traffic pairs and for anything that touches latency or the router, where the effect is large and the traffic is there.
• Guardrail metrics as independent blockers, regardless of the headline. A guardrail metric is one that must not move, whatever the headline metric does, and that blocks a launch on its own — it is not a tiebreaker and it is not weighed against the win. Here they are p99 latency, empty-output rate, wrong-target-language rate, and hallucination-flag rate from the serving QE model. Any of these moving blocks the launch even if copy rate went up.

A quality launch decision at 100 languages is made offline and confirmed online, not the other way round.

Assumptions in this stage.

• State out loud: an 8% baseline copy rate, a 0.5% relative effect worth detecting, a 1% experiment allocation, and 1.2B requests a day. Change any of them and the 7.36M-per-arm figure moves — but the ratio between the head and the tail, which is what the argument turns on, does not.
• Ask: whether the interactive surface even has a copy button. Three of the five online signals in the table above exist only on some surfaces, and a product without them has no behavioural metric at all.
• Load-bearing: that quality effects are concentrated in a handful of pairs rather than spread uniformly. That is what makes the global A/B a trap. If effects were uniform, the aggregate really would read out in 1.2 days and cover the tail, the offline per-pair gate would be redundant, and the whole launch process would be an ordinary online experiment.
• Also load-bearing: that side-by-side preference panels of about 200 segments give a usable signal for tail pairs. They are the only remaining decision instrument once the A/B is ruled out. If bilingual raters are unavailable for a direction — the same procurement problem as MQM — that direction ships on offline chrF alone.

Serving architecture

Now the design becomes a request path, priced line by line — and the pricing is why two lanes are mandatory rather than an optimization.

The diagram below is one request’s journey, top to bottom. Here is the same journey in steps, with the numbers that matter attached to each box.

1. Request arrives carrying text and a target language.
2. Normalize, sentence split, language ID if undeclared. That last step is a small classifier guessing the source language when the caller did not say which it is.
3. Exact cache lookup. The key is the normalized source plus the language pair plus the model version plus the formality setting. About 30% of requests hit here and return without touching a model at all.
4. Router (on a miss). It decides using three inputs: the pair’s resource tier, the input length, and whether this is document mode.
5. Lane B — the distilled lane. Roughly 60% of all traffic: short interactive requests on high-resource pairs. A 0.4B model at batch 64, beam 4, 620 requests per second per GPU.
6. Lane C — the MoE lane. Roughly 10% of all traffic: low-resource pairs, long inputs, document mode. An 8B mixture-of-experts model at batch 32 with document context, 23 requests per second per GPU. Note the 27x throughput gap against Lane B — that gap is the whole cost story in scale and cost.
7. Harness guards. Both lanes feed the same checks: numerals and named entities survived, length ratio is sane, nothing is repeating, output language is the one requested, and the reference-free quality score clears its floor.
8. Two repair paths. A numeral mismatch triggers a re-decode with the source numerals forced into the output. A quality score below floor escalates the request to Lane C, or returns the untranslated source with a low-confidence flag.
9. Write cache, then respond. Both repair paths rejoin here, so nothing skips the cache write.

flowchart TD
    REQ(["Request · text + target lang"]) --> NORM["Normalize · sentence split<br/>language ID if undeclared"]
    NORM --> CK{"Exact cache<br/>key = norm_src + pair<br/>+ model_version + formality"}
    CK -->|"hit · 30%"| OUT(["Response"])
    CK -->|miss| ROUTE{"Router<br/>pair tier · length<br/>· document mode"}
    ROUTE -->|"60% · high-resource<br/>short · interactive"| S["Lane B · 0.4B distilled<br/>batch 64 · beam 4<br/>620 req/s per GPU"]
    ROUTE -->|"10% · low-resource<br/>long · document mode"| L["Lane C · 8B MoE<br/>batch 32 · doc context<br/>23 req/s per GPU"]
    S --> GUARD{"Harness guards<br/>numeral + entity set<br/>length ratio · repetition<br/>output language · COMET-QE"}
    L --> GUARD
    GUARD -->|pass| WR["Write cache"]
    GUARD -->|"numeral mismatch"| CD["Re-decode with<br/>source numerals forced"]
    GUARD -->|"QE below floor"| ESC["Escalate to Lane C<br/>or return source with<br/>low-confidence flag"]
    CD --> WR
    ESC --> WR
    WR --> OUT

    style CK fill:#2d6a4f,color:#fff
    style ROUTE fill:#bc6c25,color:#fff
    style L fill:#9d0208,color:#fff
    style GUARD fill:#1d3557,color:#fff

Against the colour key: the exact cache is green because it answers a read without asking a model at all; the router is orange because what forces its threshold is memory bandwidth and weight residency rather than arithmetic; Lane C is red because routing into it is the one decision in this path you cannot take back — a 960 ms decode is committed the moment the request enters, and no downstream box makes it faster; the guard box is blue because it, not the model, is the authority on what ships.

The latency budget

Decompose the 300 ms budget into lines that each have a different fix, and the decomposition proves the big model cannot sit on the interactive path.

The interactive target is p95 under 300 ms, because the surface re-translates as the user types.

The table below breaks one request into stages and prices each one in both lanes. Compare the two columns row by row: every line is identical or near-identical except one. Then look at the totals.

Three of those lines need naming. TLS is transport layer security, the handshake that establishes the encrypted connection before any request bytes move; it is bundled with network time because on a cold connection it is a round trip you cannot skip. Prefill is the one parallel pass in which the model reads the entire source sentence; decode is the sequential crawl that produces the translation one token at a time, each token needing its own pass over the model.

The guard line is split in two on purpose, because they are different components with different bills. The deterministic checks are free arithmetic on strings. COMET-QE is a model, it runs on 96% of the traffic that reaches a model at all, and it therefore needs a row in the fleet table — which it gets, and which turns out to cost more than the chapter’s top-ranked cost lever saves.

Decode dominates both lanes: 72 of 140 ms, and 960 of 1,075 ms. The reason is derived in The kv cache the most important mechanism in this chapter, and it is worth restating in one line here.

Decode is strictly sequential — one pass over the model per output token — and it is memory-bandwidth-bound, meaning the chip spends its time waiting for the model’s weights to arrive from memory rather than doing arithmetic. So the floor on one decode step is:

time per step  =  bytes of weight that must be read  /  memory bandwidth
total decode   =  time per step × number of output tokens

The accelerator assumed here has 2.0 TB/s of memory bandwidth, which is 2,000 GB/s. Substituting for each lane, with 60 output tokens:

Lane B   0.4B params × 2 bytes (fp16)   =  0.8 GB of weights per step
         0.8 GB / 2,000 GB/s            =  0.0004 s  =  0.4 ms  <- the floor
         × 3 overhead                   =  1.2 ms per step
         × 60 output tokens             =  72 ms

Lane C   8B params × 2 bytes (fp16)     =  16 GB of weights per step
         16 GB / 2,000 GB/s             =  0.008 s   =  8 ms    <- the floor
         × 2 overhead                   =  16 ms per step
         × 60 output tokens             =  960 ms

The overhead multipliers (×3 and ×2) cover attention, the KV cache reads, and kernel launch costs that the pure weight-read floor ignores. The small model pays a larger multiplier because its fixed per-step overheads are a bigger fraction of a shorter step.

The ratio that falls out — 960 / 72 = 13x — is entirely a consequence of the model being 20x bigger and the overheads being different. Nothing about batching or scheduling appears in it.

Lane C cannot be on the interactive path, and no amount of batching fixes it — batching raises throughput and does nothing for the latency of a single sequence. So Lane C serves document mode (paste, upload, file) where a second is acceptable, plus low-resource pairs where the alternative is a bad translation rather than a fast one. On surfaces where text may visibly change, a “fast then refine” pattern shows Lane B immediately and swaps in Lane C when it lands.

Caching common phrases

An exact-match cache is worth 30% of traffic here, and the obvious upgrade to it is the one thing you must not build.

Translation requests are severely Zipfian, meaning a small number of inputs account for a very large share of volume while everything else is a long thin tail: greetings, interface strings, product names, and the same news headline pasted by 40,000 people in one hour. Exact-match lookup on a normalized source gets a 30% hit rate at 3 ms and zero GPU cost.

key  =  hash( nfkc_casefold(src) , src_lang , tgt_lang , model_version , formality )

nfkc_casefold is Unicode’s standard normalization plus lowercasing, so that visually identical strings written with different code points collide as they should. Every component of that key is load-bearing. model_version in particular: a cached translation outlives the bug that produced it, and a cache with no version in the key is a permanent record of every mistake you have ever shipped.

Do not build a semantic cache — one that serves a stored answer when a new query merely embeds close to an old one. A near-match source differing only in a number or a name gets the wrong translation returned with full confidence, and that is the failure this whole design exists to prevent. Exact match or nothing.

Document context versus batching

Document context is a modelling improvement that is unambiguously good for quality — and its real cost is a serving cost that lands on the languages that can least afford it.

Sentence-level translation loses real information: pronoun antecedents (which noun a “he” or “it” refers back to), formality — the German choice between the familiar du and the formal Sie, which English source text does not mark at all — terminology consistency across a document, gender agreement with an entity named three sentences earlier, and ellipsis, where a sentence leaves out a word that the previous sentence supplied.

Passing the four previous source sentences as context is worth roughly 2-4 chrF on a contrastive test set — one built so that each item has a correct and an incorrect variant differing only in the phenomenon being tested, here the discourse-sensitive cases — and roughly 0.2 chrF on an aggregate test set, because only 5-10% of sentences are discourse-sensitive at all. If you build document context and measure it with aggregate chrF you will conclude it did not work. You need a contrastive suite that isolates the phenomenon.

The costs are where the design decision lives:

1. Input grows 60 -> 300 tokens. Encoder attention is quadratic, so that term goes up ~25x; it is a small share of the total, so this is the cheap part.
2. Sequence lengths become ragged, and a batch pads to its longest member. With uniform 60-token inputs, padding waste is about 10%. With document context of 0-400 tokens depending on position in the document, waste exceeds 50% unless you bucket by length. Bucketing means the scheduler cannot take the next k requests off the queue — it must wait for enough same-bucket requests, which adds queueing latency precisely when the queue is thin, which is precisely the low-traffic language pairs. The cost of context lands hardest on the languages with the least traffic, again.
3. Live typing has no next sentence, and the preceding sentences change as the user edits — which invalidates the cache key on every keystroke.

The resolution is to use document context only in document mode. On the interactive path, carry a per-session formality and glossary tag — one extra token in the prefix, with no effect on sequence length or on batching — which recovers the du/Sie choice and the terminology consistency for approximately nothing.

Assumptions in this stage.

• State out loud: a 30% cache hit rate, 60 output tokens per request, beam width 4 in Lane B, and the per-lane latency lines in the budget table. Each is a measurement you would take, not a structural claim.
• Ask: whether the interactive surface really re-translates as the user types. That single product detail is where the 300 ms comes from and where the cache-invalidation-per-keystroke problem comes from.
• Load-bearing: that decode step time really is bandwidth-bound at roughly 1.2 ms for the small model and 16 ms for the large one. Those two numbers are the whole reason there are two lanes. If the big model decoded at small-model speed there would be one lane, no router, and no routing threshold to optimize — which would delete the largest cost lever in the chapter.
• Also load-bearing: that a semantic cache is unacceptable. That follows from the framing assumption that a confidently wrong translation is the expensive failure. On a product where a near-miss is cheap, semantic caching becomes legal and the 30% hit rate goes considerably higher.

Scale and cost

Converting the traffic figure into a GPU count and a daily bill, then ranking the levers that change it, produces the chapter’s most counter-intuitive result.

The inputs

Every number in this section is derived from the eight inputs below. State them out loud before you start dividing, because an interviewer who disagrees with one of them wants to say so before you have spent four minutes on arithmetic.

One cross-chapter warning: this is a different part from the H100-class accelerator used in 2b latency is dominated by decode and decode is sequential and Why decoder only and not an encoder decoder the load bearing argument, which has 3.3 TB/s and costs $2.50/hour. Carry the step-time formula across chapters, not the constants.

From traffic to QPS

1.2e9 requests/day  /  86,400 seconds/day  =  13,889 QPS average
13,889  ×  2.5 peak factor                 =  34,722 QPS peak

Everything from here is sized against the peak number, because a fleet that only handles the average is a fleet that is down at lunchtime.

Per-GPU throughput for each of the three components

All three figures are worth deriving rather than asserting, because an interviewer will ask where they came from. Note that the two decode lanes and the guard are computed differently, and the difference is the point.

The two decode lanes are bandwidth-bound, so throughput is batch size divided by that lane’s decode time from the latency budget, then derated:

Lane B    batch 64  /  0.072 s   =  889 req/s raw
          889 × 0.7 derate       =  622 req/s per GPU,  call it 620

Lane C    batch 32  /  0.960 s   =  33.3 req/s raw
          33.3 × 0.7 derate      =  23 req/s per GPU

The QE guard is not a decode at all. It reads the source and the hypothesis in one encoder forward pass and emits a single score, so it is bounded by arithmetic rather than by bandwidth, and it is derived from FLOPs. The rule of thumb is that a forward pass costs about 2 FLOPs per parameter per token:

work per request   2 × 0.55e9 params × 120 tokens (60 src + 60 hyp)
                                       =  132e9 FLOP  =  132 GFLOP

chip throughput    150 TFLOP/s × 0.7 derate
                                       =  105 TFLOP/s effective

requests per GPU   105e12 / 132e9      =  795 req/s per GPU

The fleet table

Each row below takes the peak 34,722 QPS, multiplies by that component’s share of traffic, divides by its per-GPU throughput, and then multiplies by 1.5 for regional redundancy plus one spare replica. The last column is the one you commit to.

Two rows deserve their arithmetic spelled out.

Lane C   34,722 × 0.10  =  3,472 QPS
         3,472 / 23     =  151 GPUs  ->  × 1.5  =  227

QE guard  misses reaching a model = 34,722 × 0.70  =  24,306 QPS
          of those, 96% pass the deterministic checks and pay for QE:
          24,306 × 0.96  =  23,333 QPS
          23,333 / 795   =  29.4 GPUs  ->  × 1.5  =  44

The QE row is the one most versions of this answer leave out, and it is not small. The serving diagram puts COMET-QE inside the guard box, and the hallucination guard says it is paid for on the 96% of traffic that passes the deterministic checks. A model that runs on nearly every request needs a row in the fleet table like any other.

The bill

Multiply GPUs by $48 per GPU-day:

decode fleet     51 (Lane B) + 227 (Lane C)  =  278 GPUs
                 278 × 24 h × $2.00          =  $13,344/day
QE guard          44 × 24 h × $2.00          =  $ 2,112/day
                                                -----------
total            322 GPUs                       $15,456/day

per year         $15,456 × 365               =  $5.64M/year
per million req  $15,456 / 1,200 M requests  =  $12.88 per million requests

Now the ratio the whole optimization roadmap hangs on. Lane C is 10% of traffic, and:

227 Lane C GPUs / 278 decode GPUs  =  0.817  =  82% of the decode fleet

10% of traffic is 82% of the decode fleet. That points somewhere non-obvious, which the next subsection prices.

Which lever, in dollars

Five ways to make the system cheaper, priced against each other — and the ranking is the opposite of the order people propose them in.

Before the table: the $/day column is a change against the 278-GPU decode baseline of $13,344/day, not a total fleet cost. That is why every row but the last is negative. Every row is fleet change × $48 per GPU-day, so you can check each one in your head.

Work the top row so the rest are checkable:

2 points of peak traffic  =  34,722 × 0.02  =  694 QPS

leaves Lane C:   694 / 23   =  30.2 GPUs  ->  × 1.5  =  45 fewer
arrives Lane B:  694 / 620  =   1.1 GPUs  ->  × 1.5  =   2 more
net                                                     -43 GPUs
                                          -43 × $48  =  -$2,064/day

The last row is the one that gets misquoted. +$62,736 is the addition of 1,307 GPUs to the 278-GPU decode fleet, so serving everything on Lane C is 278 + 1,307 = 1,585 decode GPUs at 1,585 × $48 = $76,080/day — that row added to the $13,344 decode baseline, not to the $15,456 total.

Now the comparison that matters:

router lever   -$2,064/day  for 2 points of traffic moved
cache lever      -$384/day  for 10 points of hit rate

headline ratio        $2,064 / $384        =  5.4x
per point of traffic  $1,032 / $38.40      =  27x

The routing threshold is 5.4x the lever the cache is — and 27x per point of traffic moved. The cache is the one everybody proposes first. Say this plainly: caching is a latency feature that also saves a little money; the router is the cost architecture.

And then say the uncomfortable thing, because it is the strongest line available here. Put the QE guard’s own cost next to the best lever:

QE guard, total cost               $2,112/day
top-ranked lever in the table      $2,064/day saved

The component the chapter nearly forgot to size outranks the lever the chapter tells you to optimize. And the cheapest way to act on it is not a routing change at all. Halve the guard model:

0.55B encoder -> 0.28B encoder    throughput roughly doubles
44 GPUs -> 23 GPUs                21 GPUs × $48  =  $1,008/day

That $1,008/day would slot fourth in the five-row lever table — behind the beam cut, the router, and int8 quantization, but 2.6x the cache lever everybody proposes first ($1,008 / $384). It is bought with a weaker quality signal on exactly the tail pairs that need it most. Size your guards before you rank your levers.

Two honest caveats to volunteer:

• Moving traffic out of Lane C is a quality decision, not a cost decision. The 2 points you move are the marginal requests where the small model is nearly as good. Finding them is a calibration problem — you need a cheap predictor of “will the small model suffice”, and the same COMET-QE model that guards hallucination can be run on Lane B’s output to escalate the ones that fail. That makes the escalation measured, which is the only version of this that is safe.
• The beam-width lever is the largest single number on the list and it is the first one to be wrong about. Beam 4 -> 2 costs roughly 0.6 chrF on high-resource pairs and considerably more on low-resource ones, where the model is less certain and the search matters more. Halving the beam saves the most money in exactly the lane where it costs the most quality.

Sanity check against price

Compare the serving cost to the list price of the same service, and the ratio explains where the money in a translation business actually goes.

Commercial machine translation lists at around $20 per million characters. An average request is 60 tokens, and a token is roughly 4 characters, so about 240 characters per request:

1M requests × 240 chars       =  240M characters
240M / 1M × $20               =  $4,800 of list price

our marginal serving cost     =  $12.88   (from the fleet arithmetic above)

$4,800 / $12.88               =  373x

The gap is not margin on the serving call. It is data acquisition, mining compute, training runs, the MQM budget, and the 96% of language directions that will never carry enough traffic to pay for themselves. The price of a translation API is set by the long tail, not by the request in front of you — and a candidate who says that has understood the business shape of the system.

Assumptions in this stage.

• State out loud: 1.2B requests a day, 60 tokens in and 60 out, a peak factor of 2.5, a 1.5x multiplier for regional redundancy and one spare replica, an A100-class part at $2.00/GPU-hour, and a 0.7 utilization derate. All six are inputs to a product of terms, so any one of them moves the $15,456 proportionally and none of them changes the shape.
• Ask: the peak factor. It is a property of how the user base is distributed across time zones, it multiplies the entire fleet, and it is the one number here you genuinely cannot guess.
• Load-bearing: the 60/10 traffic split between the two lanes. The 82%-of-the-decode-fleet-from-10%-of-traffic result — and therefore the claim that the router is the cost architecture and the cache is not — is a direct consequence of it. At a 90/10 split the conclusion strengthens; at 30/40 the big lane is the system, and the router stops being a lever and becomes a rationing mechanism.
• Also load-bearing: that a calibrated predictor of “the small model will suffice” can be built. Moving traffic from the big lane to the small one is the top lever, and without that predictor the move is a blind quality cut rather than a measured trade.

Failure modes

Four failures are worth deriving in full, and six more fit in a table. Each gets a concrete trace, the mechanism that produces it, and the specific control that catches it — because “we would add guardrails” is not an answer.

1. Hallucinated content — the dangerous one

First, the failure the whole design exists to prevent: a fluent, confident sentence that says something the source did not.

Two traces below. The first is the subtle form — a single dropped negation. Look at which of the three checks catches it, because only one does.

SOURCE  (de)   "Der Termin wurde nicht verschoben."
HYP     (en)   "The appointment has been postponed."
                                     ^^^^^^^^^^^^^ negation dropped

chrF vs reference          0.71   -> passes any surface-metric gate
COMET-QE (reference-free)  0.42   -> below the 0.60 floor  -> CAUGHT
cross-attention entropy    normal -> not caught by the attention guard

The German says the appointment was not postponed. The English says it was. nicht is one token, and dropping it inverts the meaning while leaving the sentence perfectly fluent — which is why chrF, a surface-overlap score, still gives it 0.71.

Now the pure form, under domain shift. Here the output has nothing to do with the input at all, and the cheap checks catch it easily:

SOURCE  (is)   "th th th th"                         (noise, or an unsupported script)
HYP     (en)   "The Lord bless you and keep you."

cross-attention mass on source tokens   0.09   -> diffuse  -> CAUGHT
length ratio                            8.0    -> outside [0.5, 2.0]  -> CAUGHT

Cross-attention in those traces is the mechanism by which each output token looks back at the source tokens, and the mass is how much of that looking was aimed at real source words rather than at padding or the end-of-sentence marker. Diffuse mass means the output was not really conditioned on the input.

Mechanism, and it is the objective from the second section. The loss lives on the target side, so nothing trains the model to depend on the source. When the encoder provides no usable signal — noise, an unsupported script, a domain the model never saw — the decoder’s language model takes over and emits the mode of its training distribution. For low-resource languages that distribution is dominated by religious text, because Bible and Watchtower translations are the only large parallel corpora that exist for many languages. The hallucination is not random; it is the training corpus’s most probable sentence, which is why it is so fluent.

The design runs three guards in order of cost:

Be precise about what the ordering buys, because the obvious claim — “the expensive one is only paid for when the free ones have passed” — reads as a saving, and the chapter’s own number says it is not much of one. The ordering saves the QE call on about 4% of traffic. (That is where the “96% of traffic reaching a model” line in the fleet table comes from.)

Its real purpose is different: a deterministic hit is auditable and a model score is not. When H01 or H02 fires, you can point at the length ratio or the attention mass in the trace and a reviewer can check it. “The quality model said 0.41” is a number with no argument attached. Cheapest-first is an explainability order that happens to shave 4% off the guard’s bill.

The code below implements exactly that ordering. Read hallucination_guards first; the assertions underneath replay the two traces above and count how often the QE model is called, because “the model is only paid for when the free checks pass” is a claim about call counts and should be tested as one.

def hallucination_guards(src, hyp, attn, qe_model, pair_cfg):
    """Ordered cheapest-first. Deterministic checks before any model call."""
    flags = []

    ratio = len(hyp) / max(len(src), 1)
    lo, hi = pair_cfg.len_ratio
    if not lo <= ratio <= hi:
        flags.append(("H01 length ratio", ratio))

    # Cross-attention mass the decoder placed on real source tokens (not EOS
    # or padding). Diffuse mass means the output is not conditioned on input.
    src_mass = attn.mean_mass_on_source_tokens()
    if src_mass < 0.25:
        flags.append(("H02 output not conditioned on source", src_mass))

    if flags:                       # deterministic hit: no need to pay for QE
        return flags

    score = qe_model.score(src, hyp)          # reference-free COMET-QE
    if score < pair_cfg.qe_floor:             # per-pair, calibrated on MQM
        flags.append(("H03 QE below per-pair floor", score))
    return flags


# --- the two traces above, plus the claim the ordering actually makes --------
class _Cfg2:
    len_ratio, qe_floor = (0.5, 2.0), 0.60


class _Attn:
    def __init__(self, mass):
        self._m = mass

    def mean_mass_on_source_tokens(self):
        return self._m


class _QE:
    """Counts its own calls, because 'the model is only paid for when the free
    checks pass' is a claim about how often it is called."""

    def __init__(self, score):
        self.score_value, self.calls = score, 0

    def score(self, src, hyp):
        self.calls += 1
        return self.score_value


# Dropped negation: normal length, normal attention, caught only by QE.
_qe = _QE(0.42)
_f = hallucination_guards("Der Termin wurde nicht verschoben.",
                          "The appointment has been postponed.",
                          _Attn(0.55), _qe, _Cfg2())
assert [c for c, _ in _f] == ["H03 QE below per-pair floor"], _f
assert _qe.calls == 1                     # deterministic checks passed -> QE paid

# Pure hallucination under domain shift: length ratio AND attention both fire.
_qe = _QE(0.42)
_f = hallucination_guards("th th th th", "The Lord bless you and keep you.",
                          _Attn(0.09), _qe, _Cfg2())
assert [c for c, _ in _f] == ["H01 length ratio",
                              "H02 output not conditioned on source"], _f
assert _qe.calls == 0, "a deterministic hit must short-circuit the model call"

# A clean translation pays for QE and passes it.
_qe = _QE(0.81)
assert hallucination_guards("Der Termin wurde verschoben.",
                            "The appointment has been postponed.",
                            _Attn(0.61), _qe, _Cfg2()) == []
assert _qe.calls == 1
print("hallucination guards: dropped negation caught by QE only, "
      "pure hallucination caught deterministically without paying for QE")

Note what those assertions pin down. The dropped negation has a normal length ratio and normal attention mass, so the only layer that catches it is the model — which is why the ordering cannot be sold as a saving. The pure hallucination fires two deterministic checks and the QE model is never called, which is the 4% the ordering does save. And the per-pair floor is calibrated against MQM rather than guessed.

2. Gender bias

Some failures cannot be fixed in the model at all, because the information the model would need is genuinely not in its input.

SOURCE  (tr)   "O bir doktor. O bir hemsire."       ('o' is genderless in Turkish)
HYP     (en)   "He is a doctor. She is a nurse."

Mechanism: the target language forces a distinction the source does not encode. The model must emit something, and it resolves the ambiguity with the training corpus’s prior over occupation-gender co-occurrence. This is not a decoding bug and it is not fixable by “debiasing the model,” because the information genuinely is not in the source.

So it is a product decision. Detect ambiguity — score the hypothesis with the pronoun swapped; if the two scores differ by less than a threshold, the source did not determine it — and emit both alternatives with a note. The failure is not that the model guessed; it is that the interface presented a guess as a translation.

3. Named-entity and numeral mangling

Next, the failure that is cheapest to catch and most embarrassing to ship — and the fix is code rather than a model.

SOURCE  (en)   "Contact Dr. Nguyen at 555-0142, room 3B, by March 4."
HYP     (fr)   "Contactez le Dr Nguyen au 555-0124, salle 3B, avant le 3 mars."
                                          ^^^^^^^^                    ^^^^^^^
                                          digits transposed           wrong date

Mechanism: numerals fragment into low-information subword pieces (Tokens) and the decoder’s language model has no preference between 0142 and 0124. Both are equally plausible continuations. There is no signal in the objective that makes one win.

This is a harness control — code that runs on the output before it ships — not a prompt trick and not a training fix.

Three things about it are load-bearing, and all three are things a one-line regular expression cannot do:

1. It compares anchors, not just a sorted multiset of values. {555-0142, 555-0143} on both sides is a match as a multiset even when Alice’s number has been handed to Bob.
2. It splits on non-digits rather than letting a character class span separators. A greedy class turns Rooms 1, 23 into the single number 123, and Salles 12, 3 into 123 as well, so a mangled pair compares equal.
3. It parses dates in each locale’s own order. 3/4 is March 4 in en-US and 3 April in fr, so reading both sides the same way makes a real error invisible.

The code below is the guard. Read the three module-level constants first (the locale sets and the number-word table), then numerals(), then numeral_guard(), which runs the three checks in order and returns a code plus the repair it wants. The assertions at the bottom are the four inputs a naive version got wrong.

import re

# Locales that write 1.234,5 where English writes 1,234.5. Defined here rather
# than imported: a guard whose constants live in someone else's module raises
# NameError the first time it meets a German price.
DECIMAL_COMMA_LOCALES = {"de", "fr", "es", "it", "pt", "nl", "pl", "ru", "tr", "is"}

# Locales that write a bare numeric date day-first. This is why `3/4` is not
# one date: en-US reads March 4, everything below reads 3 April.
DAY_FIRST_LOCALES = DECIMAL_COMMA_LOCALES | {"en-GB"}

# Numbers a correct translation may spell out instead of leaving as digits.
# Without this the guard fires on every "3 rooms" -> "trois salles", and the
# stated repair -- force-copy the source digits -- would push digits into a
# translation that was already right.
NUMBER_WORDS = {
    "en": {"zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
           "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
           "ten": "10", "eleven": "11", "twelve": "12"},
    "fr": {"zéro": "0", "un": "1", "une": "1", "deux": "2", "trois": "3",
           "quatre": "4", "cinq": "5", "six": "6", "sept": "7", "huit": "8",
           "neuf": "9", "dix": "10", "onze": "11", "douze": "12"},
    "de": {"null": "0", "eins": "1", "ein": "1", "eine": "1", "zwei": "2",
           "drei": "3", "vier": "4", "fünf": "5", "sechs": "6", "sieben": "7",
           "acht": "8", "neun": "9", "zehn": "10", "elf": "11", "zwölf": "12"},
}

# A separator may appear only BETWEEN digit runs. The one-line version this
# replaced, `\d[\d.,\s]*\d`, spanned whitespace and swallowed "1, 23" into the
# single token "123".
NUM_PAT = r"\d+(?:[.,]\d+)*"
WORD_PAT = r"[^\W\d_]+"
TOKEN = re.compile(f"{NUM_PAT}|{WORD_PAT}", re.UNICODE)
# The separator is a named piece so that the source of this block never
# contains the character sequence "](", which markdown link checkers read
# as a link.
SEP = r"[/.\-]"
DATE = re.compile(r"\b(\d{1,2})" + SEP + r"(\d{1,2})(?:" + SEP + r"(\d{2,4}))?\b")


def _canon(raw: str, locale: str) -> str:
    """1.234,5 (de) and 1,234.5 (en) are the same number."""
    raw = raw.replace(".", "").replace(",", ".") \
        if locale in DECIMAL_COMMA_LOCALES else raw.replace(",", "")
    return raw.rstrip(".") or "0"


def dates(text: str, locale: str) -> list[tuple]:
    """Numeric dates as (year, month, day), read in the locale's own order."""
    out = []
    for m in DATE.finditer(text):
        a, b, y = int(m.group(1)), int(m.group(2)), m.group(3)
        day, month = (a, b) if locale in DAY_FIRST_LOCALES else (b, a)
        if 1 <= month <= 12 and 1 <= day <= 31:
            out.append((int(y) if y else None, month, day))
    return sorted(out, key=lambda t: (t[0] is None, t))


def numerals(text: str, locale: str) -> list[tuple[str, str]]:
    """(anchor, value) pairs in reading order. The anchor is the nearest
    preceding capitalised word — a cheap stand-in for the entity the number
    belongs to, and the part of a sentence most likely to survive translation
    unchanged, which is what makes it comparable across the two sides."""
    text = DATE.sub(" ", text)                  # dates are checked separately
    spelled = NUMBER_WORDS.get(locale.split("-")[0], {})
    out, anchor = [], ""
    for m in TOKEN.finditer(text):
        tok = m.group()
        if tok[0].isdigit():
            out.append((anchor, _canon(tok, locale)))
        elif tok.lower() in spelled:
            out.append((anchor, spelled[tok.lower()]))
        elif tok[:1].isupper() and m.start() > 0:   # skip the sentence-initial word
            anchor = tok
    return out


def numeral_guard(src, hyp, src_locale, tgt_locale):
    """Three checks, none of them a bare set comparison, all of them before the
    response ships. Returns None when the hypothesis is clean."""
    s, h = numerals(src, src_locale), numerals(hyp, tgt_locale)

    # 1. Multiset of VALUES. A digit invented, dropped or mistyped shows here.
    if sorted(v for _, v in s) != sorted(v for _, v in h):
        return {"code": "N01 numeral value set differs",
                "source": [v for _, v in s], "hypothesis": [v for _, v in h],
                "action": "re-decode with source numerals force-copied"}

    # 2. Anchored comparison. Values can match as a multiset while being
    # re-assigned across entities — Alice's number handed to Bob. Only anchors
    # appearing verbatim on BOTH sides are comparable across languages.
    for a in sorted({a for a, _ in s if a} & {a for a, _ in h if a}):
        if sorted(v for k, v in s if k == a) != sorted(v for k, v in h if k == a):
            return {"code": "N02 numerals re-assigned across entities",
                    "entity": a,
                    "source": sorted(v for k, v in s if k == a),
                    "hypothesis": sorted(v for k, v in h if k == a),
                    "action": "re-decode with the entity-number binding constrained"}

    # 3. Dates, each side read in its own order. `3/4` is not one day.
    ds, dh = dates(src, src_locale), dates(hyp, tgt_locale)
    if ds != dh:
        return {"code": "N03 date differs once locale order is applied",
                "source": ds, "hypothesis": dh,
                "action": "re-decode with the date rendered unambiguously"}
    return None


# --- the four inputs the sorted()-multiset version shipped wrong ------------
assert numeral_guard("Call Alice at 555-0142 and Bob at 555-0143.",
                     "Appelez Alice au 555-0143 et Bob au 555-0142.",
                     "en", "fr") is not None          # numbers swapped: was PASS
assert numeral_guard("Rooms 1, 23 are open.",
                     "Salles 12, 3 sont ouvertes.",
                     "en", "fr") is not None          # regex glued "1, 23": was PASS
assert numeral_guard("Deliver on 3/4/2024.",
                     "Livraison le 3/4/2024.",
                     "en", "fr") is not None          # 4 Mar vs 3 Apr: was PASS
assert numeral_guard("There are 3 rooms.",
                     "Il y a trois salles.",
                     "en", "fr") is None              # spelled out: was a false N01

# --- and it still passes correct translations, and still catches the chapter's
# own trace from the top of this subsection
assert numeral_guard("Deliver on 3/4/2024.", "Livraison le 4/3/2024.",
                     "en", "fr") is None
assert numeral_guard("Call Alice at 555-0142 and Bob at 555-0143.",
                     "Appelez Alice au 555-0142 et Bob au 555-0143.",
                     "en", "fr") is None
assert numeral_guard("Der Preis ist 1.234,5 EUR.", "The price is 1,234.5 EUR.",
                     "de", "en") is None
assert numeral_guard(
    "Contact Dr. Nguyen at 555-0142, room 3B, by March 4.",
    "Contactez le Dr Nguyen au 555-0124, salle 3B, avant le 3 mars.",
    "en", "fr")["code"].startswith("N01")
print("numeral guard: 8/8 assertions passed")

Each code carries its own repair, and they are not the same repair. On N01 — a value invented, dropped or mistyped — re-decode with the source numerals forced into the output by constrained decoding: at each step, restrict the model’s choice of next token to those that keep the output valid, so a wrong digit is not unlikely but unreachable. On N02 the values are all present and merely attached to the wrong entity, so force-copying them changes nothing; the constraint has to be on the binding, which is the harder decode. On N03 the digits are identical on both sides and only the reading differs, so force-copying them actively preserves the bug — the repair is to render the date unambiguously (4 March 2024) rather than to copy 3/4.

The spelled-out case is why the naive version of this guard is worse than no guard. "There are 3 rooms." translated as "Il y a trois salles." is correct, and a guard that compares digit sets fires on it and then applies N01’s repair — forcing the digit 3 into a translation that had already spelled the number out properly. A guard whose false positive is repaired by corrupting a correct output is a guard that ships damage. That is the specific reason the number-word table is in the code and not in a backlog ticket.

4. Degenerate repetition

One failure the training objective never saw, because it is created by the search procedure used at serving time rather than by the model.

SOURCE  (km)   [a 120-token legal sentence with no clear clause boundaries]
HYP     (en)   "...the party shall be liable for the party shall be liable
                for the party shall be liable for the party shall be lia"
                                                    ^ truncated at max length

Mechanism: beam search’s length-normalized score can make a loop locally optimal. Once "the party shall be liable for" is in the prefix, the highest-probability continuation given that identical context is the same sequence again — each repetition raises the conditional probability of the next repetition. It is the same self-reinforcing dynamic as an agent loop (Why loops self reinforce), and it fires on the same trigger: an input the model has no good response to.

Three guards, all needed. N-gram blocking during decode bans any four-token run that this hypothesis has already emitted, which breaks the loop mechanically. A coverage penalty added to the beam score punishes candidates that left source tokens with no attention on them, which is what a looping output does. And the length-ratio check from the hallucination guard catches whatever the first two miss.

5. The rest

The remaining six failures fit in one table, each with the signature you would see in a trace and the control that catches it.

Assumptions in this stage.

• State out loud: the guard thresholds — a length ratio band of 0.5 to 2.0, cross-attention mass below 0.25, and a per-pair quality floor calibrated on MQM — plus the claim that the two deterministic checks fire often enough that the quality model is paid for on about 96% of traffic rather than 100%.
• Ask: whether returning the untranslated source with a low-confidence flag is acceptable product behaviour. It is the only safe fallback when a guard fires and no re-decode helps, and some surfaces cannot display it.
• Load-bearing: that hallucination is detectable at serving time without a reference translation. Every control in this section is a runtime check. If none of them separated a hallucination from a good translation, the only remaining designs are human review — priced at 24M reviews a day and rejected — or refusing to serve the directions where the risk is highest.
• Also load-bearing: that cross-attention mass is available and meaningful. It is free because the decoder already computed it, which is what makes the cheapest-first ordering affordable. On an architecture that does not expose it, the guard stack loses its middle layer and the quality model has to run on every request.

Alternatives considered and rejected

Each design below was considered, and each is paired with the number that killed it — because rejecting on a number rather than a preference is what the round is actually scoring. Every entry in the third column is either a figure derived earlier in the chapter or a mechanism you can state in one sentence.

Two linguistic terms in the pivot row need unpacking, because they are what makes that rejection stronger than an arithmetic one. The T-V distinction is a grammatical split between familiar and formal second-person address — French tu against vous, German du against Sie — which English does not have. Evidentiality is a grammatical marker, obligatory in Turkish and many other languages, indicating how the speaker came by the information: witnessed directly, or heard from someone else. Neither survives a trip through English, because English has no slot to put them in. That is the difference between a pivot and a noisy channel: a noisy channel corrupts information, and a pivot deletes categories of it outright. The arithmetic — 0.87 adequacy per leg composing to 0.76 — understates the damage, because it treats the loss as uniform when it is concentrated exactly on the features the source language marked and the pivot language does not.

Interviewer pushback

Eleven questions this design actually gets asked, what each one is testing, and an answer that survives the follow-up. Almost every one of them is an attack on an assumption rather than on a fact.

“Why does back-translation work at all? You’re training on machine output.” Testing: whether you know where the gradient is. Because the machine output is on the input side, and the loss only touches target tokens. Synthetic source is input noise, which regularizes the encoder; the target is real human text, which trains the decoder’s language model — the component a small parallel corpus starves. Forward-translation puts machine output on the target side, where the gradient is, so the model learns to imitate its own errors. Measured on our en->is set, forward-translation raised chrF by 0.6 and raised the hallucination rate from 4.1% to 5.6%. And decode the back-translations with sampling, not beam — beam output is the low-entropy mode and the model overfits to it.

“One model for 100 languages, or 100 models?” Testing: whether you cost the serving side or only the training side. One, and the argument is residency rather than quality. 9,900 pairwise models is 7,920 GB of weights, 114 GPUs to hold before serving a request, and 9,850 of those pairs at 0.28 QPS each — 0.045% utilization on hardware that does not get cheaper when idle. A multilingual model is O(1) in pair count and every GPU can serve every request, so the scheduler packs freely. Then quality follows: zero-shot covers the 9,800 directions that have no data at all. The cost is capacity dilution, which is real and is why I would size the parameter budget by language count rather than by dataset size.

“So add another 30 languages?” Testing: whether you understood the curse or just named it. Only with more parameters. My 400M model has 262M in the embedding table, leaving 138M of body; if half of that is effectively language-private, going from 25 to 100 languages already cut per-language private capacity from 2.76M to 0.69M params. Adding 30 more on a fixed budget pays for them out of the low-resource languages I already have. Languages with close relatives in the set partly offset it via transfer; an isolate takes the full loss. The scalable answer is MoE — total parameters grow, active parameters per token do not — but that costs memory residency, so it belongs in the big lane only.

“Your BLEU went up 1.2 points. Ship it?” Testing: whether you take a metric at face value. It is a trap. No, and BLEU alone would not be enough even if it went up 12. Take “the meeting was postponed until Thursday” as the reference. The correct paraphrase — delayed for postponed — scores 0.000, because the substitution lands at token 4 of 7 — dead centre, inside all four 4-gram windows — so p_4 = 0 and the geometric mean collapses. The factual inversion — Tuesday for Thursday — scores 0.643, because token 6 sits near the end where only two 4-grams reach it and the leading ones survive. Both are one token away; BLEU prefers the wrong one, and the thing deciding the score is where the error landed. chrF2 compresses the gap to 0.72 against 0.84 but still ranks them the same way round. It is also not comparable across languages: p_n is computed over target-language tokens, so a morphologically rich language loses up to four n-grams to a single wrong inflection and has a structurally lower ceiling, while Chinese and Japanese scores are a property of your segmenter. And it is not comparable across tokenizations, so I would want a sacreBLEU signature before believing the 1.2 at all. What actually gates the launch is the per-pair table — no directed pair may drop more than 1.0 chrF or 0.02 COMET — plus the guardrails: hallucination flag rate, wrong-target-language rate, empty output.

“How would you catch a fluent translation that says the opposite of the source?” Testing: whether you have a serving-time control or only an offline metric. Three layers, cheapest first, and only the third involves a model. Length ratio against a per-pair calibrated band. Cross-attention mass on real source tokens — below 0.25 means the output is not conditioned on the input, which is the pure hallucination signature and it is free because the decoder already computed it. Then reference-free COMET-QE with a per-pair floor calibrated against MQM, which is the layer that catches the dropped negation, since that one has normal length and normal attention. Surface metrics catch none of these; chrF on the dropped-negation example is 0.71.

“What is your actual cost driver?” Testing: whether you have done the arithmetic. The routing threshold. 322 GPUs at $15,456/day, of which 278 are the two decode lanes and 44 are the QE guard; Lane C is 10% of traffic and 82% of that decode fleet, because an 8B model does 23 requests per second per GPU against Lane B’s 620. So moving 2 points of traffic from C to B is worth $2,064/day and raising the cache hit rate 10 points is worth $384 — a 5.4x difference off five times fewer points of traffic, so 27x per point moved, and the cache is what everyone proposes first. The honest caveat is that moving traffic out of C is a quality decision. You need a calibrated predictor of “the small model suffices,” and the QE model I already run for hallucination is exactly that predictor.

“Why not use an LLM for translation? They’re better.” Testing: whether you can say yes and still reject it. They are better, particularly on document-level context, register, and instruction-following, and I would use one — as a teacher. Sequence-level distillation from an 8B or larger teacher into a 0.4B student beats training the student on the real data, because the real data’s diversity exceeds what 0.4B can represent. But on the serving path it is 20-50x the cost and 10x the latency against a 300 ms budget, and the student inherits the teacher’s hallucinations, so stage 5’s regression gate has to test for those specifically rather than just chrF.

“Document-level translation is strictly better. Why isn’t it the default?” Testing: whether you see the systems consequence of a modeling choice. Because it breaks batching. Uniform 60-token inputs pad to about 10% waste; document context of 0-400 tokens pushes that past 50% unless you bucket by length, and bucketing means the scheduler waits for enough same-bucket requests instead of taking the next k off the queue. That queueing cost lands hardest where the queue is thinnest, which is the low-traffic pairs — the same ones the tokenizer already treats worst. Also, on a live-typing surface there is no next sentence and the preceding ones change on every keystroke, invalidating the cache. So: document context in document mode, and a one-token formality/glossary tag on the interactive path, which recovers the du/Sie and terminology wins without touching sequence length.

“Your Amharic quality is bad. What do you do first?” Testing: whether you look at the tokenizer. Measure fertility before touching the model. Amharic sits at 5.2 tokens per word against English’s 1.15, because the vocabulary was fit on a corpus that is 40% English and Amharic is 0.05% of it. That single number explains three symptoms at once: 4.5x the cost, 4.5x the decode steps — which is 4.5x the 72 ms of decode in a 140 ms budget, so ~392 ms end to end against a 300 ms p95 — and a model composing meaning from character fragments it has seen in far fewer contexts. The fixes are temperature-sampling the vocabulary-fitting corpus with the same T as the training mix, which most teams forget, and flooring each language’s piece allocation. Only after that would I add data — because more data through a bad tokenizer buys less than it should.

“You have no parallel data for Icelandic to Swahili. What ships?” Testing: whether you can start. Zero-shot from the multilingual model, with the target-language tag as the first decoder token rather than a source prefix so it cannot be attended away, plus a wrong-language penalty in the beam and an output-language classifier that triggers a re-decode. Expect 20-50% off-target rate before those guards and low single digits after. Then improve it with synthetic data: back-translate Swahili monolingual text into Icelandic through the model itself, which is noisy but creates a pair that did not exist. And gate the whole thing on a side-by-side preference panel with bilingual raters at 200 segments, because an online A/B on that pair would need 3.4 years to reach power.

“How do you know a launch didn’t break a language nobody on your team speaks?” Testing: whether your evaluation scales to the tail. It cannot be the aggregate metric and it cannot be an online A/B — both are structurally blind to a pair carrying 0.05% of traffic. It is a fixed 500-segment set per directed pair with per-pair chrF and COMET thresholds as a hard blocker, plus guardrail metrics that fire per pair: wrong-target-language rate, hallucination flag rate from the serving QE model, empty-output rate, and p99 latency. Then MQM monthly on a rotating subset of 12 pairs, which is 300 annotator-hours — $12,000, and about four days on a ten-person panel — so it is a launch gate rather than a CI gate. The design principle is that the tail is protected offline and confirmed online, never the reverse.

The assumption ledger

Every assumption the chapter has leaned on, collected in one place, so you can state the design’s foundations in twenty seconds and say what replaces the design when each one fails.

The bins are the same three used at the end of every section above, and the same three used in Assumptions are the design:

• State it — you are free to pick, and being wrong costs a re-derivation.
• Ask it — the answer changes the architecture, so it is worth an interviewer’s time.
• Load-bearing — if it is wrong the design is not suboptimal, it is invalid.

The rows are sorted load-bearing first. The last column is the one to rehearse: it is what you say when an interviewer knocks an assumption out.

The sentence that makes this visible to an interviewer: “This design rests on three things. One, a 300 ms p95 budget, which is what forces two lanes and therefore creates the router that is the entire cost architecture. Two, that back-translation’s noise lands on the input side where no gradient flows — that single asymmetry is what lets me serve 9,800 language directions that have no data. Three, that I can detect a hallucination at serving time without a reference translation, because if I cannot, the only honest alternatives are human review at 24 million reviews a day or refusing to serve the languages that need it most.”

Cheat sheet

The one-line answers to the questions this design is most often asked. Everything here is derived above; this is the recall test.

Next: 04 — Assistant Chatbot.