How to design a machine-translation system

Machine translation (MT) takes text written in one language and produces text in another that means the same thing.

In this lesson, we’ll design that system at the scale where the interesting problems live: 100 languages and a billion requests a day. At that size three problems dominate the design, and none of them is the model architecture:

1. Data. You cannot collect parallel corpora for 9,900 language directions; you have to manufacture them.
2. Tokenization. A single choice made before training sets both cost and quality for the languages you have the least data for.
3. Routing. Where you draw the line between a small fast model and a large slow one decides most of the bill.

We’ll work through each, plus how to measure translation quality (harder than it looks) and where the money actually goes. By the end you’ll be able to justify manufacturing training data instead of collecting it, defend a single shared model against a per-pair fleet, and name the one tokenizer decision that sets cost, latency, and quality together.

Problem framing

• Input: a span of text, a target language, and sometimes a declared source language.
• Output: text in the target language that means what the source meant.
• Scale and constraints: 100 languages, about 1.2 billion requests a day, and p95 under 300 ms on the interactive surface. Traffic is heavily skewed: the top 50 directed pairs are roughly 80% of volume; the remaining ~9,850 are a long tail nobody can afford to serve well one at a time.
• Why it is hard: the failure that matters is not a clumsy sentence. It is a fluent sentence that asserts something the source did not, because no reader who lacks the source language can detect it.

A directed pair is ordered: English-to-Icelandic and Icelandic-to-English are different problems. So 100 languages give 100 × 99 = 9,900 directed pairs.

Three instincts worth overturning before the details, each derived below:

• Data: you do not collect parallel corpora, you manufacture them, by mining pairs out of web crawls and by back-translation.
• Model count: one model per pair is killed by serving cost, not training cost. A single shared multilingual model replaces the fleet.
• Metric: raising BLEU is not the same as raising quality. BLEU can score a factual inversion above a correct paraphrase.

The training objective (why hallucination is built in)

The system is a conditional sequence model, p(y | x, L_tgt): the probability of an output sentence y given a source sentence x and a tag L_tgt naming the target language. Training maximizes the log-likelihood of the human reference, which is the same as minimizing the average per-token loss:

loss  =  - (1/|y|) · sum over t of  log p( y_t | y_<t, x, L_tgt )

Here y_t is the correct next target word, y_<t the target words before it, x the whole source, and L_tgt the target-language tag; dividing by |y| averages per token instead of summing over the sentence. A model that puts 0.9 on the right word pays -log(0.9) = 0.11; one that puts 0.1 on it pays 2.30. Confidently wrong is expensive; confidently right is nearly free.

Two consequences drive the rest of the lesson:

• The loss lives only on the target side. The sum runs over target tokens, never over the source. Nothing in the objective forces the model to depend on the source. The decoder, the half of the model that emits output one word at a time, is a language model that has been offered a conditioning signal from the encoder, the half that reads the source. When that signal is uninformative (an out-of-domain source, a garbled source, a language the encoder barely represents) the highest-likelihood output is whatever the decoder’s own language model prefers. Every hallucination failure in this lesson follows from that one fact.

• Decoding is a separate decision from training. Training scores one token at a time; serving has to search for a high-scoring whole sequence. The default is beam search: keep the k best partial translations at every step and extend all of them instead of greedily committing to the single best next word, with scores length-normalized so short outputs do not win by default. Beam search introduces a failure the training objective never saw, degenerate repetition, covered below.

Data: manufacture, don’t collect

Very little real parallel data exists, so two techniques manufacture the rest: mining translation pairs out of web crawls, and back-translation.

The supply gap

Demand is 9,900 directed pairs. Almost all published parallel data has English on one side, so the supply is the English-centric pairs: 99 into English plus 99 out of English is 198 directed pairs, about 2% of demand, and public corpora cover maybe 100 of those at useful volume.

The last row is the point: about 98% of directed pairs have essentially no real training data, and roughly 9,800 of 9,900 have no usable corpus at all. Two terms used from here on: zero-shot means translating a direction the model was never trained on; mined bitext means pairs discovered inside a web crawl instead of published as a corpus.

The data pipeline

flowchart TD
    CR["Web crawl<br/>documents per language"] --> SEG["Sentence split<br/>+ language ID both sides"]
    SEG --> EMB["Multilingual sentence encoder<br/>embed every sentence"]
    EMB --> IDX[("ANN index<br/>per language")]
    IDX --> MINE["Nearest-neighbour mining<br/>margin criterion"]
    MINE --> FILT{"Quality filters<br/>langid · length ratio<br/>near-dup · MT-detector<br/>entity + numeral agreement"}
    FILT -->|"reject 60-80%"| DROP["Discard"]
    FILT -->|keep| PAR[("Mined parallel<br/>corpus")]
    MONO[("Monolingual<br/>target text")] --> BT["Reverse model T->S<br/>sampled decoding"]
    BT --> SYN[("Synthetic pairs<br/>noisy source · REAL target")]
    PAR --> MIX["Temperature-sampled mix<br/>q_p proportional to D_p^(1/T)"]
    SYN --> MIX
    MIX --> TRAIN["Train S->T"]
    TRAIN -.->|"round 2 seeds a better<br/>reverse model"| BT

The mining leg turns a crawl into pairs: split into sentences and re-run language ID per sentence (a crawl labelled “Icelandic” is full of English); embed every sentence with a multilingual encoder so that sentences with similar meaning land near each other; index the embeddings in an ANN (approximate nearest neighbour) structure that finds close vectors without comparing against all of them, which is what makes search affordable at billions of sentences; mine nearest neighbours with the margin criterion below; and pass the candidates through a filter bank that discards 60-80% of them. What survives is a mined parallel corpus.

The back-translation leg manufactures pairs from monolingual text: run monolingual target text backwards through a reverse translation model with sampled decoding, producing synthetic pairs whose source side is noisy machine output and whose target side is real human text.

The mixture combines mined and synthetic pairs by temperature sampling (below). The forward model trained on the mixture seeds a better reverse model for the next round, the dotted loop.

Mining: why raw cosine fails

Closeness between a sentence and a candidate translation is measured by cosine similarity, which is 1.0 when two vectors point the same way and 0 when unrelated. Ranking by raw cosine fails for a structural reason. Some sentences are hubs ("Yes.", "Click here.", "Copyright 2019.") that carry so little content they sit near the centroid (the average position of all vectors) and are therefore near everything. Rank by raw cosine and the top matches are hubs paired with unrelated hubs, in every language.

The fix normalizes a pair’s cosine by local density, the margin criterion:

margin(x, y)  =  cos(x, y)  /  mean over the k nearest neighbours of x and y of cos(., .)

  a genuine pair:   cos 0.82,  local mean 0.51  ->  margin 1.61
  a hub-hub pair:   cos 0.79,  local mean 0.77  ->  margin 1.03

The margin asks “is this pair unusually close for these two sentences”, which raw cosine cannot. Threshold around 1.06, trading recall for precision.

Filtering: precision over recall

A misaligned pair is not a neutral zero-information example. It is a gradient step teaching the model “produce fluent target text only loosely related to this source”, the hallucination failure, taught deliberately. On a 500M-pair corpus, a 2% misalignment rate is 10 million such examples, which is more supervised signal than most low-resource pairs have in total. A small misalignment rate is not a small quality tax; it sets a hallucination floor no serving guard can remove. So the filter bank is precision-first and discards most of its input.

Each filter is a reject rule applied at ingest:

• Language ID mismatch on either side.
• Length ratio outside a per-pair band (roughly 0.5-2.0).
• Alignment margin below threshold.
• Numeral sets differ across sides: numbers must survive translation. The comparison canonicalizes notation first, so 1.234,5 (German) and 1,234.5 (English) count as equal; the filter must fire on missing or changed values, not on formatting.
• Target side looks machine-translated (a classifier). The web is now full of MT output; training on it entrenches your own past errors as ground truth and drags every language toward translationese, grammatical prose that reads as translated, with the source language’s word order and idiom showing through.
• Near-duplicate of an existing pair (via MinHash fingerprints, which let you find near-copies without comparing every pair). Crawls are pathologically duplicated, and duplicated pairs get memorized instead of learned, which is how a model comes to emit a Bible verse when given noise.

Back-translation: put the noise where the gradient isn’t

Say you have 2M English-Icelandic pairs and 800M monolingual Icelandic sentences, 400x more monolingual data, apparently useless because it has no source side. Manufacture one: train a reverse Icelandic-to-English model on whatever real data exists, run it over the monolingual Icelandic to get synthetic English, then train the forward English-to-Icelandic model on those (synthetic English, real Icelandic) pairs.

Why it works is the target-side loss again. The gradient, the signal telling each weight which way to move, flows only through terms in the loss, which are the target tokens. The synthetic side is the source x: errors there are input noise, which regularizes the encoder (teaches it to cope with imperfect input and generalize better). The target y is real human Icelandic, so the decoder’s language model, the component a small parallel corpus starves, trains on genuine fluent text.

Forward-translation reverses this: translating monolingual English through your own model puts machine output on the target side, where the gradient is, so the model learns to imitate its own output, its own translationese and systematic errors. It is worse than doing nothing.

Back-translation puts the noise where the gradient is not; forward-translation puts it where the gradient is.

Three practical points:

1. Decode back-translations with sampling, not beam search. Beam output is the mode of the distribution, clean, low in variety, and far less varied than real source text. Train on it and the model overfits a narrow synthetic-input distribution and then degrades on real input. Sampled output has realistic variety. This is the most commonly skipped step.
2. Cap the synthetic ratio around 3:1. Past that, the target distribution is dominated by whatever monolingual domain you crawled, and the model drifts.
3. Iterate, but stop at about three rounds. Each round is worth roughly a third of the last.

Measured on an English-to-Icelandic development set (chrF is a character n-gram overlap score, COMET is a learned quality score from 0 to 1, and hallucination rate is the fraction of outputs asserting something the source did not say):

The second row makes the point: forward-translation raised a surface metric slightly and made hallucination worse, exactly as the asymmetry predicts.

The data mix: temperature sampling

If training batches are drawn in proportion to corpus size, a pair with a thousand times less data essentially never appears. Flatten the distribution: sample pair p in proportion to D_p^(1/T), where D_p is its corpus size and T a temperature. T = 1 keeps natural proportions; larger T pulls everything toward uniform.

On the real extremes (English-French at 2B pairs against English-Icelandic at 2M, a 1000:1 ratio) T = 5 gives 1000^(1/5) = 10^0.6 ≈ 3.98 : 1. A fifth root turns three orders of magnitude into a factor of four.

The cost is the mirror image. English-French has 1000x the sentences but is sampled only about 4x as often, so each individual English-Icelandic sentence is shown roughly 250x more per epoch. At ~250 repetitions a sentence risks being memorized instead of learned, which is why near-duplicate rejection and a per-pair repeat cap are what make the temperature safe.

Model choice: one multilingual model

One specialist model per language pair fails on serving cost, not quality.

Why the pairwise fleet dies: weight residency

A model can only answer a request if its weights are sitting in GPU memory at that moment, and you rent that memory by the hour whether or not a request arrives. This is weight residency, and it does not amortize over traffic.

Back-of-envelope: 9,900 pairs at 0.4B parameters each, 2 bytes per parameter in fp16 (16-bit floats), is about 7,920 GB of weights against roughly 70 GB usable per GPU, about 114 GPUs just to hold the weights before serving a single request. Overlay traffic: the tail’s ~9,850 pairs share 20% of ~13,900 QPS, about 0.28 QPS per pair, against ~620 QPS a GPU can serve, roughly 0.045% utilization. You rent a chip and use one part in 2,200 of it.

Loading a model only on demand does not save it either. A 0.8 GB model read from NVMe (fast local solid-state storage) at 3 GB/s is about 270 ms of cold-start latency, which nearly consumes the 300 ms budget before the first token, and at 0.28 QPS requests almost never arrive back to back, so nearly every request pays it.

One multilingual model is O(1) in the number of pairs: one set of weights serves all 9,900 directions, every GPU can serve every request, and the batch scheduler can pack requests freely instead of only combining ones that happen to want the same pair.

Zero-shot transfer, and its failure

A model trained only on English-to-everything and everything-to-English can translate Icelandic into Swahili without ever seeing that pair. The encoder maps source text into a largely language-agnostic representation, so same-meaning sentences land near each other, and the target-language tag tells the decoder which language to write. Quality is below a supervised model and well above nothing, and nothing is the honest alternative for 9,800 directions.

The characteristic failure is off-target translation: asked for Icelandic-to-Swahili, the model emits English, because English was the dominant target in training and the tag is a single token against that prior. Untreated rates of 20-50% are normal. Fixes, cheapest first:

• Put the target-language tag as the first token the decoder emits, so it cannot be attended away, free.
• Add a wrong-language penalty from a cheap language-ID classifier inside beam search, about 2% of decode time.
• Reject and re-decode at serve time when the output language disagrees with the request, one extra decode on 1-3% of tail traffic.
• Add non-English-centric synthetic data by back-translating one non-English language into another through the model itself.

The curse of multilinguality

Adding languages to a fixed parameter budget helps low-resource languages through transfer (a language borrowing structure from its relatives) up to a point, then hurts everything. The parameters are a fixed pot: some shared across languages, some effectively spent on one language’s quirks, and the second pot gets divided more ways as you add languages.

Back-of-envelope on a 400M model: about 262M is the shared embedding table (below), leaving ~138M of transformer body; treat roughly half as effectively language-private, so ~69M. That is 2.76M per language at 25 languages and 0.69M at 100, a 4x cut, which is just 100 / 25. Transfer partly offsets this for languages with close relatives; a language isolate like Basque (no known relatives) borrows nothing and takes the full hit.

So “should we add 30 more languages?” has one honest answer: yes, if you also grow the parameter budget; otherwise you pay for them with the low-resource languages you already have. Two ways to buy per-language capacity at constant per-token compute (compute counted in FLOPs, floating-point operations):

• Mixture of experts (MoE): many parallel copies of some layers (the experts) with a small router sending each token to one or two. Total parameters grow while parameters used per token stay fixed. The cost is residency: every expert must sit in memory though most are idle for any given token, so MoE only pays where you can afford the memory. Hence: MoE in the big lane, a dense distilled model in the throughput lane.
• Language-specific adapters: small per-language weight patches on a shared trunk. Rejected for a serving reason given below: a batch can only hold requests that share weights.

Tokenization: one decision, three costs

The design is one shared subword vocabulary across all 100 languages. A subword vocabulary is a fixed list of text fragments (whole common words, prefixes, suffixes, single characters) and tokenizing greedily covers input with pieces from that list. It is fit once, before training, and cannot be changed afterwards without retraining from scratch. (See the tokens section of the LLM internals chapter for the mechanism.)

Shared, not per-language, for three reasons:

1. Transfer flows through shared tokens. Cognates, shared scripts, digits, punctuation, and named entities occupy the same embedding rows across languages. With disjoint vocabularies, Information in English and German would be two unrelated rows and nothing would transfer.
2. Zero-shot depends on it. The shared representation is anchored by tokens that appear in more than one language.
3. One embedding matrix instead of 100, which is also the problem.

The vocabulary is most of the model

The first thing a model does with a token is look it up in an embedding matrix: one row per vocabulary entry, d_model numbers wide, where d_model is the width of every internal representation. Its parameter count is just vocab × d_model.

vocab 256,000  ×  d_model 1,024   =  262M params

A model normally has two such tables, one turning input tokens into vectors, one turning the final vector into a score per vocabulary entry. Tying them (using one matrix for both) halves that. Tied, the table is 262M of a 400M model, about two-thirds; untied it would be 524M, larger than the whole model.

So in a small multilingual model the vocabulary is the majority of the parameters, not a preprocessing detail. Three consequences: tie the input and output embeddings (mandatory at this size); treat vocabulary size as a serving-memory decision, reviewed like the number of layers; and consider a factorized embedding, a narrow 256,000 × 128 table multiplied back up by a 128 × 1,024 matrix, about 8x fewer parameters, at a cost of ~0.8 chrF on high-resource pairs because every token’s identity now squeezes through 128 numbers instead of 1,024.

Script imbalance and fertility

The vocabulary is fit on a corpus, and the algorithm spends its fixed budget of pieces where the text is. If the corpus is 40% English, the pieces go on English fragments; a language at 0.05% of the corpus gets almost none and falls back toward single characters or raw bytes.

Fertility measures this: the average number of tokens the tokenizer produces per word of a language. Fertility 1.0 is one token per word; 5.2 means five pieces to write one word.

Turkish is agglutinative: it builds long words by gluing suffixes onto a stem, so a whole English clause can be one Turkish word that a tokenizer without dedicated suffix-pieces shatters. Ge’ez is the script Amharic is written in, and its several hundred characters are essentially absent from an English-dominated corpus.

The three costs compound instead of trading off, all from Amharic’s 4.5x fertility:

• Cost. The same sentence costs 4.5x more tokens. Users pay per character, so they never see it; your GPU bill sees nothing else.
• Latency. Decode is one sequential step per token, so 4.5x the tokens is 4.5x the decode. In Lane B’s 140 ms budget, decode is 72 ms and everything else is fixed, so 68 + 4.5 × 72 ≈ 392 ms, over the 300 ms p95, on a request the model handles no differently.
• Quality. A 60-word sentence is ~312 tokens in Amharic against ~69 in English, and attention cost (every token looking at every other) grows with the square of length, so about (312/69)² ≈ 20x the attention compute, on top of the model having to compose meaning from fragments it has seen in far fewer contexts.

The worst-quality languages also get the worst latency and cost, all from one decision made when the tokenizer was fit. That makes tokenization a fairness and capacity decision, not a preprocessing step. Three fixes: temperature-sample the vocabulary-fitting corpus with the same T used for training (most teams sample the training mix but fit the vocabulary on the raw one); floor each language’s piece allocation so no script is squeezed out; and track fertility per language as a first-class metric, so a vocabulary change cannot silently regress one language while improving the average.

Training

Five stages run in order. Stage 1 takes almost all the compute; stages 3-5 are cheap and decide most of the shipped quality.

flowchart LR
    S1["1. Pretrain body<br/>temperature-sampled mix<br/>most of the compute"] --> S2["2. Iterative<br/>back-translation<br/>2-3 rounds"]
    S2 --> S3["3. Fine-tune<br/>clean in-domain pairs"]
    S3 --> S4["4. Distil throughput model<br/>8B teacher to 0.4B student"]
    S4 --> S5["5. Quantize + calibrate<br/>int8, per-pair gate"]

Three terms: distillation trains a small “student” to imitate a large “teacher”, so the student inherits behaviour it was too small to learn directly. Quantization stores weights in a smaller number format (here int8, 8-bit integers) halving the bytes read per decode step and roughly doubling decode speed at some accuracy cost. A held-out set is data kept out of training so measuring on it is not measuring memorization.

Stage 4 has one wrinkle. Sequence-level distillation trains the student on the whole sentence the teacher produced, not on the teacher’s raw per-token scores. This is the back-translation asymmetry read in reverse: the machine output now lands on the target side, where the gradient is, so the student learns the teacher’s mode. That mode is cleaner and less varied than real data, so it is easier for 0.4B parameters to fit than the real data’s full diversity, and the student beats a 0.4B model trained directly on real data. It also inherits the teacher’s hallucinations, so the stage-5 gate must test for those, not just chrF.

Measuring quality

The launch decision runs on a ladder of measurements. Start with the limits of the metric everyone quotes.

BLEU, and what it cannot see

BLEU compares a candidate against a human reference by counting shared n-grams (runs of n consecutive tokens). It is the geometric mean of four clipped precisions (n = 1..4) with a length correction:

BLEU  =  BP · exp( sum over n=1..4 of (1/4) · log p_n )

p_n is the fraction of candidate n-grams that also appear in the reference, clipped so a candidate cannot get credit for repeating an n-gram more often than the reference contains it. Being a geometric mean, a single zero precision makes the whole score zero. BP, the brevity penalty, scales the score down when the candidate is shorter than the reference, a patch that exists only because precision alone is maximized by emitting one confident word and stopping. So BLEU is a string-overlap statistic with a length correction bolted on.

Worked example. One reference, two candidates, each one word off:

REFERENCE               The meeting was postponed until Thursday .
A (meaning preserved)   The meeting was delayed   until Thursday .
B (meaning destroyed)   The meeting was postponed until Tuesday  .

Both candidates are 7 tokens, so BP = 1 and all the action is in the precisions. In A, delayed sits at position 4 (dead centre, inside all four 4-gram windows) so p_4 = 0 and the geometric mean collapses. In B, Tuesday sits at position 6, near the end, where only two 4-grams reach it and the leading ones survive.

BLEU ranks the factual inversion above the correct paraphrase, and the mechanism is positional, not semantic: a substitution’s cost is set by how many 4-gram windows cover its position. Repeat the same one-word error at each position and BLEU traces a triangle peaking at the centre:

Unigram precision is 6/7 in every row and tells the seven cases apart not at all; everything that moves is window arithmetic. A system tuned to maximize BLEU is being paid to move its errors toward the ends of sentences. Three further limits, often stated wrongly:

• Not comparable across languages. p_n is over target-language tokens, so a morphologically rich language loses up to four n-grams to one wrong inflection and has a structurally lower ceiling; Chinese and Japanese have no whitespace, so the score depends on your segmenter. A BLEU of 32 on en-de and 32 on en-fi are not the same quality.
• Not comparable across tokenizations. Identical output detokenized differently scores differently. sacreBLEU fixes the tokenization and emits a signature recording the settings; a BLEU number without a signature is a claim, not a measurement.
• Not sensitive where it matters. Modern systems cluster tightly and a 0.3 BLEU gap is inside reference-translator noise. Any reported difference needs paired bootstrap resampling (repeatedly resample the test set and see how often the winner changes); most differences do not survive it.

The metric ladder

Arrange the rest by how often you can afford to run them: the trustworthy ones are slow, the fast ones shallow.

flowchart LR
    subgraph CHEAP["Per commit · seconds · deterministic"]
        CHRF["chrF2<br/>character n-gram F-score<br/>beta=2, recall-weighted"]
    end
    subgraph MID["Per release · minutes · GPU"]
        COMET["COMET / BLEURT<br/>learned regressor on<br/>human judgements"]
        QE["COMET-QE<br/>reference-free<br/>deployable at serve time"]
    end
    subgraph SLOW["Pre-launch · days · money"]
        MQM["MQM human eval<br/>error spans + severities"]
    end
    CHRF --> COMET --> MQM
    COMET --> QE
    MQM -.->|"retrains / recalibrates"| COMET

• chrF2 is an F-score over character n-grams up to length 6, weighting recall twice as heavily as precision (beta = 2). Averaging over n = 1..6 instead of taking a geometric mean means no zero cliff. It degrades gracefully with morphology (a wrong Finnish inflection loses a suffix’s characters, not a whole word plus four n-grams) and needs no word tokenizer, which removes BLEU’s cross-tokenization problem. Its limit: it has no positional or semantic term and ranks purely by character overlap. On the Thursday/Tuesday example it compresses the spread (0.72 for A against 0.84 for B) but still ranks the inversion higher, for a different reason than BLEU: Thursday and Tuesday share T, u, s and day, while postponed and delayed share almost nothing. chrF fixes the morphology problem, not the meaning problem; no surface-overlap statistic can, because the information is not in the surface.
• COMET and BLEURT are learned metrics: a multilingual encoder reads the source, the hypothesis, and the reference, and a small regression head predicts the human score. They agree with human sentence-level judgement at roughly 0.4-0.6 Kendall tau (rank correlation) against ~0.2 for BLEU, and they do separate Thursday from Tuesday. Caveats: scores are comparable only within one version of the metric model (pin it, like a sacreBLEU signature); optimizing against them is reward hacking (minimum-Bayes-risk decoding against COMET produces outputs that score high and read strangely, so keep the decode metric separate from the reported one); and they inherit their annotators’ biases. The reference-free variant, COMET-QE, is the deployable one, because production has no reference translation to compare against. It is the single most useful model in the serving path.
• MQM (multidimensional quality metrics) is what quality actually means: professional bilingual annotators mark error spans by category (accuracy (mistranslation, omission, addition; fluency) grammar, register, terminology) and severity, scored as a negative weighted error count per 100 words. It produces diagnostics (“60% of the regression is addition errors”) that a single number cannot. It costs roughly $12,000 and a few days per eval round, so it is launch-blocking on a fixed set, not per-commit.

The gate that protects the tail

Use per-pair regression gates, not an aggregate. At 100 languages a pair carrying 0.05% of traffic can go from good to garbage and move aggregate chrF by 0.003, inside the noise. So the gate is stated per direction: no directed pair may drop more than 1.0 chrF or 0.02 COMET against the current production model, measured on a fixed 500-segment set for that pair. An aggregate improvement buys no exemption.

Launch decisions: why the tail can’t be A/B tested

Live traffic offers behavioural signals: copy/share rate, re-translation rate (a strong negative signal), edit rate (highest quality, editor surfaces only), language-swap rate (a language-ID failure, not a translation one, so attribute it correctly), downstream conversion (confounded by everything). But the launch decision for the lowest-traffic languages cannot be an online experiment at all.

An A/B test needs a sample size that grows as the effect you want to detect shrinks. Back-of-envelope: detecting a 0.5% relative change in an 8% copy rate needs about 7.4M requests per arm. At a 1% experiment allocation that is ~1.2 days globally, which is the trap, because the effect is concentrated in a handful of pairs, not spread uniformly. Slice by pair and each slice needs its own 7.4M; a pair at 0.05% of traffic sees only ~6,000 requests a day in the experiment, so powering it would take about 3.4 years. That is structural, not a tuning problem.

So the launch decision is:

• Offline per-pair regression gates as the blocker (above), the real gate.
• Side-by-side preference panels for the tail: bilingual raters compare two outputs. Far more power per observation (usable at ~200 segments), but it measures preference, not correctness.
• Online A/B for high-traffic pairs and anything touching latency or the router, where the effect is large and the traffic is there.
• Guardrail metrics as independent blockers, regardless of the headline: p99 latency, empty-output rate, wrong-target-language rate, and hallucination-flag rate. Any one moving blocks the launch on its own.

Quality at 100 languages is decided offline and confirmed online, not the reverse.

Serving

Two lanes, an exact cache, and a guard layer. One request’s path:

1. Request arrives with text and a target language.
2. Normalize, sentence split, language ID if the source was not declared.
3. Exact cache lookup. About 30% of requests hit here and return without touching a model.
4. Router (on a miss), deciding on the pair’s resource tier, input length, and whether this is document mode.
5. Lane B, the distilled lane. Roughly 60% of traffic: short interactive requests on high-resource pairs. A 0.4B model at ~620 requests/second/GPU.
6. Lane C, the MoE lane. Roughly 10% of traffic: low-resource pairs, long inputs, document mode. An 8B mixture-of-experts model at ~23 requests/second/GPU, a 27x throughput gap that is the whole cost story below.
7. Harness guards on both lanes: numerals and entities survived, length ratio is sane, nothing repeats, output language matches the request, and the reference-free quality score clears its floor.
8. Repair paths. A numeral mismatch triggers a re-decode with source numerals forced; a below-floor quality score escalates to Lane C or returns the untranslated source with a low-confidence flag.
9. Write cache, then respond.

flowchart TD
    REQ(["Request · text + target lang"]) --> NORM["Normalize · sentence split<br/>language ID if undeclared"]
    NORM --> CK{"Exact cache<br/>key = norm_src + pair<br/>+ model_version + formality"}
    CK -->|"hit · 30%"| OUT(["Response"])
    CK -->|miss| ROUTE{"Router<br/>pair tier · length<br/>· document mode"}
    ROUTE -->|"60% · high-resource<br/>short · interactive"| S["Lane B · 0.4B distilled<br/>620 req/s per GPU"]
    ROUTE -->|"10% · low-resource<br/>long · document mode"| L["Lane C · 8B MoE<br/>23 req/s per GPU"]
    S --> GUARD{"Harness guards<br/>numeral + entity set<br/>length ratio · repetition<br/>output language · COMET-QE"}
    L --> GUARD
    GUARD -->|pass| WR["Write cache"]
    GUARD -->|"numeral mismatch"| CD["Re-decode with<br/>source numerals forced"]
    GUARD -->|"QE below floor"| ESC["Escalate to Lane C<br/>or return source with<br/>low-confidence flag"]
    CD --> WR
    ESC --> WR
    WR --> OUT

The latency budget

The interactive target is p95 under 300 ms, because the surface re-translates as the user types. Broken into stages, priced in both lanes:

TLS is the handshake that establishes the encrypted connection, bundled with network time because on a cold connection it is a round trip you cannot skip. Prefill is the one parallel pass in which the model reads the whole source; decode is the sequential crawl producing the translation one token at a time.

Decode dominates both lanes and is the only line that differs much. Decode is strictly sequential (one pass over the model per output token) and memory-bandwidth-bound, meaning the chip spends its time waiting for weights to arrive from memory instead of doing arithmetic. So per step, time = bytes of weights read / memory bandwidth. At 2.0 TB/s: Lane B’s 0.4B params at 2 bytes is 0.8 GB, about a 0.4 ms floor, ~1.2 ms with overhead, times 60 tokens is 72 ms; Lane C’s 8B is 16 GB, an 8 ms floor, ~16 ms with overhead, times 60 is 960 ms. The 13x gap is the model being 20x bigger; batching does not touch it, because batching raises throughput and does nothing for a single sequence’s latency. (See the KV-cache section of the LLM internals chapter.)

So Lane C cannot sit on the interactive path. It serves document mode (paste, upload, file) where a second is acceptable, plus low-resource pairs where the alternative is a bad translation instead of a fast one. On surfaces where text may visibly change, a “fast then refine” pattern shows Lane B immediately and swaps in Lane C when it lands.

Caching

Translation requests are heavily Zipfian: a few inputs (greetings, interface strings, product names, the same headline pasted by 40,000 people in an hour) are a large share of volume, with a long thin tail behind them. Exact-match lookup on a normalized source gets a ~30% hit rate at ~3 ms and zero GPU cost. The key is:

key  =  hash( nfkc_casefold(src) , src_lang , tgt_lang , model_version , formality )

where nfkc_casefold is Unicode normalization plus lowercasing, so visually identical strings collide. model_version is load-bearing: a cached translation outlives the bug that produced it, so a cache with no version in the key is a permanent record of every mistake you shipped.

Do not build a semantic cache, one that serves a stored answer when a new query merely embeds close to an old one. A near-match differing only in a number or name gets the wrong translation returned with full confidence, the exact failure this design exists to prevent. Exact match or nothing.

Document context versus batching

Sentence-level translation loses real information: pronoun antecedents, formality (German du versus Sie, unmarked in English source), terminology consistency, gender agreement with an entity named sentences earlier, and ellipsis. Passing the previous four source sentences is worth roughly 2-4 chrF on a contrastive test set (each item a correct/incorrect pair differing only in the phenomenon), but only ~0.2 chrF on an aggregate set, because only 5-10% of sentences are discourse-sensitive. Measure document context with aggregate chrF and you will conclude it did not work.

Its real cost is serving, and it lands on the languages that can least afford it. Document context makes sequence lengths ragged (0-400 tokens), and a batch pads to its longest member, so padding waste exceeds 50% unless you bucket by length, and bucketing makes the scheduler wait for enough same-bucket requests, adding queueing latency exactly where the queue is thinnest, the low-traffic pairs. Also, live typing has no next sentence, and the preceding sentences change on every keystroke, invalidating the cache.

Resolution: document context only in document mode. On the interactive path, carry a per-session formality and glossary tag (one extra prefix token, with no effect on sequence length or batching) which recovers du/Sie and terminology consistency for almost nothing.

Scale and cost

Convert traffic to a fleet and a bill, then rank the levers. The inputs: 1.2B requests/day; 60 source and 60 target tokens on average; a peak factor of 2.5; an A100-class GPU with 70 GB usable and 2.0 TB/s of bandwidth at $2.00/GPU-hour ($48/GPU-day); and a 0.7 utilization derate.

Traffic to QPS: 1.2B / 86,400 s ≈ 13,889 QPS average, times 2.5 is ~34,722 peak. Everything is sized against peak, because a fleet that only handles the average is down at lunchtime.

Per-GPU throughput: the two decode lanes are bandwidth-bound, so throughput is batch size over decode time, derated. Lane B ~620 req/s, Lane C ~23 req/s. The QE guard is a single encoder forward pass, so it is compute-bound (roughly 2 FLOPs per parameter per token) and does ~795 req/s.

The fleet takes peak QPS times each component’s traffic share, over its per-GPU throughput, times 1.5 for regional redundancy plus one spare:

The QE-guard row is the one most sizings forget: COMET-QE runs on ~96% of the requests that reach a model, so it needs a fleet row like any other model. The bill: 322 × $48 ≈ $15,456/day, about $5.64M/year, or ~$12.88 per million requests.

The result the whole roadmap hangs on: Lane C is 10% of traffic but 227 of the 278 decode GPUs, about 82% of the decode fleet, because an 8B model does 23 req/s against Lane B’s 620.

Which lever, in dollars

Each $/day below is a change against the 278-GPU decode baseline, so all but the last are negative:

The routing threshold is a 5.4x larger lever than the cache, and 27x per point of traffic moved. Caching is a latency feature that saves a little money; the router is the cost architecture. Two honest caveats: moving traffic out of Lane C is a quality decision, the 2 points you move are the marginal requests where the small model is nearly as good, which you find with the same COMET-QE model you already run, making the escalation measured, not blind; and the beam-width lever, the largest single number, is the first to be wrong about, because halving the beam saves the most money in exactly the low-resource lane where it costs the most quality.

The QE guard’s own cost, $2,112/day, outranks the top routing lever on its own. The cheapest way to act on it is not a routing change but halving the guard model (0.55B to 0.28B encoder, ~$1,008/day saved), bought with a weaker quality signal on exactly the tail pairs that need it most. Size your guards before you rank your levers.

Sanity check against price. Commercial MT lists around $20 per million characters; at ~240 characters per request that is ~$4,800 of list price per million requests against a ~$12.88 marginal serving cost, a ~373x gap. That gap is not margin on the serving call. It is data acquisition, mining compute, training runs, the MQM budget, and the 96% of directions that will never carry enough traffic to pay for themselves. The price of a translation API is set by the long tail, not by the request in front of you.

Failure modes

Hallucination — the dangerous one

A fluent, confident sentence that says something the source did not. Two forms.

The subtle form, a dropped negation:

SOURCE (de)  "Der Termin wurde nicht verschoben."   (the appointment was NOT postponed)
HYP    (en)  "The appointment has been postponed."

chrF vs reference          0.71   -> passes any surface-metric gate
COMET-QE (reference-free)  0.42   -> below the 0.60 floor  -> CAUGHT
cross-attention            normal -> not caught by the attention guard

Only the quality model catches this. The pure form, under domain shift, the cheap checks catch:

SOURCE (is)  "th th th th"                        (noise, or an unsupported script)
HYP    (en)  "The Lord bless you and keep you."

cross-attention mass on source tokens  0.09  -> diffuse           -> CAUGHT
length ratio                           8.0   -> outside [0.5,2.0] -> CAUGHT

Cross-attention is the mechanism by which each output token looks back at the source tokens; the mass is how much of that looking landed on real source words instead of padding. Diffuse mass means the output was not really conditioned on the input.

Mechanism: the target-side loss again. When the encoder provides no usable signal, the decoder’s language model emits the mode of its training distribution, which for low-resource languages is dominated by religious text, since Bible and Watchtower translations are the only large parallel corpora many languages have. The hallucination is fluent because it is the training corpus’s most probable sentence.

Three guards, cheapest first:

The ordering saves the QE call on only about 4% of traffic. That is where the “96% reaching a model” figure comes from, so it is not really a cost saving. Its real purpose is explainability: a deterministic hit is auditable (“length ratio 8.0”, “attention mass 0.09”) while “the quality model said 0.41” has no argument attached. The dropped negation has normal length and attention, so only the model catches it, which is why the ordering cannot be sold as a saving.

Gender bias — not fixable in the model

SOURCE (tr)  "O bir doktor. O bir hemsire."   ('o' is genderless in Turkish)
HYP    (en)  "He is a doctor. She is a nurse."

The target language forces a distinction the source does not encode, so the model must emit something and resolves it with the training corpus’s occupation-gender prior. This is not a decoding bug and is not fixable by debiasing, because the information genuinely is not in the source. So it is a product decision: detect ambiguity (score the hypothesis with the pronoun swapped; if the two scores are close, the source did not determine it) and emit both alternatives with a note. The failure is not that the model guessed; it is that the interface presented a guess as a translation.

Numeral and entity mangling — code, not a model

SOURCE (en)  "Contact Dr. Nguyen at 555-0142, room 3B, by March 4."
HYP    (fr)  "...au 555-0124, salle 3B, avant le 3 mars."   (digits transposed, wrong date)

Numerals fragment into low-information subword pieces, and the decoder’s language model has no preference between 0142 and 0124: both are equally plausible continuations. This is a harness control that runs on the output before it ships. Three things it must do that a one-line regex cannot:

1. Compare anchors, not just a set of values. {555-0142, 555-0143} matches as a set even when Alice’s number has been handed to Bob, so compare each entity’s numbers, anchored to the nearest preceding capitalised word (the part of a sentence most likely to survive translation).
2. Split on non-digits instead of letting a character class span separators. A greedy class reads Rooms 1, 23 as the single number 123, so a mangled pair compares equal.
3. Parse dates in each locale’s order. 3/4 is March 4 in en-US and 3 April in fr, so reading both sides the same way hides a real error.

It must also canonicalize notation (1.234,5 in German equals 1,234.5 in English) and accept spelled-out numbers ("3 rooms" to "trois salles"), or it will fire on correct translations and then “repair” them by forcing digits into output that was already right, a false positive that ships damage, which is why a naive digit-set guard is worse than no guard.

Each finding carries its own repair. An invented, dropped, or mistyped value triggers a re-decode with the source numerals force-copied via constrained decoding: at each step, restrict the token choice so a wrong digit is unreachable. A value merely re-assigned to the wrong entity needs the entity-number binding constrained instead. A date whose digits match but reading differs is rendered unambiguously (4 March 2024) instead of copied.

Degenerate repetition — created by the search, not the model

SOURCE (km)  [a long legal sentence with no clear clause boundaries]
HYP    (en)  "...the party shall be liable for the party shall be liable for the party shall be lia"
                                                  ^ truncated at max length

Beam search’s length-normalized score can make a loop locally optimal: once "the party shall be liable for" is in the prefix, the highest-probability continuation is the same sequence again, and each repetition raises the probability of the next. It is the same self-reinforcing dynamic as an agent loop (see why loops self-reinforce), triggered by an input the model has no good response to. Three guards, all needed: n-gram blocking during decode bans any four-token run already emitted, breaking the loop mechanically; a coverage penalty in the beam score punishes candidates that left source tokens with no attention, which is what a looping output does; and the length-ratio check catches whatever the first two miss.

The rest

Alternatives considered and rejected

Two terms make the pivot rejection stronger than the arithmetic. The T-V distinction is the familiar/formal split in second-person address (French tu versus vous), which English lacks. Evidentiality is a grammatical marker, obligatory in Turkish and many languages, indicating whether the speaker witnessed something directly or heard it secondhand. Neither survives a trip through English, because English has no slot for them. That is the difference between a pivot and a noisy channel: a noisy channel corrupts information; a pivot deletes categories of it outright, and the loss is concentrated exactly on the features the source marked and the pivot language does not.

Final design

flowchart TD
    subgraph DATA["Data manufacture"]
        CRAWL["Web crawl + monolingual text"] --> MINEBT["Mining (margin) + back-translation<br/>+ precision-first filters"]
        MINEBT --> MIX["Temperature-sampled mix (T=5)"]
    end
    subgraph TRAIN["Training"]
        MIX --> PRE["Pretrain 8B multilingual body<br/>shared 256k vocabulary"]
        PRE --> FT["Back-translation rounds + clean fine-tune"]
        FT --> DIST["Distil 0.4B throughput student<br/>+ int8 quantize"]
    end
    subgraph SERVE["Serving · p95 < 300 ms"]
        REQ(["Request"]) --> CACHE{"Exact cache · 30%"}
        CACHE -->|miss| ROUTER{"Router"}
        ROUTER -->|"60% high-resource"| LB["Lane B · 0.4B · 620 req/s"]
        ROUTER -->|"10% low-resource / docs"| LC["Lane C · 8B MoE · 23 req/s"]
        LB --> G["Guards: length · attention · numerals<br/>output language · COMET-QE"]
        LC --> G
        G --> RESP(["Response"])
    end
    DIST --> LB
    FT --> LC
    G -.->|"per-pair gate + MQM"| FT

What the design depends on

A few assumptions are load-bearing: if one is wrong, the design is not merely suboptimal, it is invalid.

Three more are worth confirming before building, because the answer changes the architecture: whether traffic really is concentrated (uniform traffic weakens the residency case and makes a global A/B legitimate); whether there is legal permission to train on crawled web text (a “no” deletes the mining half and drops coverage to a few hundred directions); and the peak factor, which multiplies the entire fleet and is a property of time-zone distribution you must measure, not guess.

Conclusion

• At 100 languages, the model architecture is the least of it. The design turns on manufacturing data for directions that have none, a tokenizer decision made before training, and a routing threshold.
• Most training data does not exist and must be manufactured: mined from crawls with the margin criterion and precision-first filters, and back-translated. Back-translation works because the loss lives only on the target side, so machine noise on the source side regularizes while real human text on the target side trains the decoder.
• One shared multilingual model beats a per-pair fleet on serving cost, because weight residency does not amortize; the price is capacity dilution, which is a parameter-budget question, not a linguistics one.
• One tokenizer decision sets cost, latency, and quality together for a language, and hits the lowest-resource ones hardest, a fairness decision disguised as preprocessing.
• Surface metrics measure overlap, not meaning: BLEU can rank a factual inversion above a correct paraphrase purely on where the error lands. Quality is gated per-pair offline and confirmed online, never the reverse, and hallucination is caught at serve time by length, attention, and a reference-free quality model.
• The router, not the cache, is the cost architecture: 10% of traffic is 82% of the decode fleet, so moving marginal traffic to the small lane is the largest lever, and the QE guard is a fleet component in its own right, easy to forget and not small.

One line to remember: at 100 languages the model is the easy part; the design is won or lost in the data you manufacture, the tokenizer you freeze before training, and the routing threshold you pick.

Further reading

• Vaswani et al., “Attention Is All You Need” (2017), the transformer.
• Sennrich, Haddow & Birch, “Improving Neural Machine Translation Models with Monolingual Data” (2016), back-translation.
• Edunov, Ott, Auli & Grangier, “Understanding Back-Translation at Scale” (2018), why sampled decoding beats beam for back-translation.
• Artetxe & Schwenk, “Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings” (2019), the margin criterion.
• Fan et al., “Beyond English-Centric Multilingual Machine Translation” (M2M-100, 2021).
• NLLB Team, “No Language Left Behind: Scaling Human-Centered Machine Translation” (2022).
• Papineni et al., “BLEU: a Method for Automatic Evaluation of Machine Translation” (2002); Popović, “chrF” (2015); Rei et al., “COMET” (2020); Post, “A Call for Clarity in Reporting BLEU Scores” (sacreBLEU, 2018).
• Freitag et al., “Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation” (MQM, 2021).
• Kim & Rush, “Sequence-Level Knowledge Distillation” (2016).

Next: Assistant Chatbot.