How to design a machine-translation system

At 100 languages and ~1.2B requests/day, the model is the easy part; the design turns on manufacturing data, one frozen tokenizer decision, and a routing threshold.

Framing

• 100 languages give 100 × 99 = 9,900 directed pairs (order matters). Top ~50 pairs are ~80% of volume; ~9,850 are a long tail.
• Target: p95 < 300 ms interactive. The dangerous failure is a fluent sentence asserting something the source did not.
• Objective: conditional seq model p(y | x, L_tgt), loss = -(1/|y|) · Σ_t log p(y_t | y_<t, x, L_tgt). Loss lives only on the target side, so nothing forces dependence on the source. Every hallucination follows from this.

Data: manufacture, don’t collect

• ~98% of directed pairs have essentially no real data. Two techniques fill the gap: mined bitext (pairs found in web crawls) and back-translation.
• Mining ranks by margin criterion, not raw cosine: margin = cos(x,y) / mean cos over k nearest neighbours of x,y. Threshold ~1.06. Fixes hubs (low-content sentences near the centroid, near everything).
• Filter bank is precision-first, drops 60-80%: langID, length ratio 0.5-2.0, margin, numeral-set match, MT-detector, near-dup (MinHash). A 2% misalignment on 500M pairs = 10M hallucination-teaching examples; it sets a floor no serving guard removes.
• Back-translation: run reverse T→S model (sampled decoding) over monolingual target text → (noisy source, REAL target). Noise lands where the gradient is not; the real target trains the decoder’s language model. Forward-translation puts machine output on the target side and is worse than nothing.
• Sample back-translations, not beam (beam = low-variety mode). Cap synthetic ~3:1. Stop at ~3 rounds.

• Temperature sampling the mix: sample pair p ∝ D_p^(1/T). T=5 turns a 1000:1 ratio into ~4:1 (1000^(1/5)≈3.98), but shows each rare sentence ~250x more/epoch, so near-dup rejection + repeat cap are what make it safe.

One multilingual model

• Per-pair fleet dies on weight residency, not training: weights must sit in GPU memory whether or not a request arrives, and it does not amortize. 9,900 × 0.4B fp16 ≈ 7,920 GB ≈ 114 GPUs just to hold; tail runs ~0.045% utilization. On-demand load costs ~270 ms cold start on nearly every request.
• One shared model is O(1) in pairs and enables zero-shot (translate an untrained direction via language-agnostic encoder + target tag). Characteristic failure: off-target translation (emits English), untreated 20-50%. Fix cheapest-first: tag as first decoded token, langID penalty in beam, reject+re-decode, non-English synthetic data.
• Curse of multilinguality: fixed parameter budget divides more ways as languages grow (~2.76M/lang at 25 → 0.69M at 100). Buy capacity at constant per-token FLOPs via MoE (big lane, costs residency) or adapters (rejected: a batch only holds requests that share weights).

Tokenization: one decision, three costs

• One shared subword vocabulary across all languages, fit once before training, unchangeable without full retrain. Shared enables transfer + zero-shot (same embedding rows across languages).
• The vocab is most of the model: 256,000 × 1,024 = 262M of a 400M model. Tie input/output embeddings (mandatory). Optional factorized embedding (256k×128 up-projected by 128×1024) ~8x fewer params, costs ~0.8 chrF.
• Fertility = tokens per word. An English-dominated corpus starves rare scripts; the three costs compound, all from fertility.

• 4.5x fertility → 4.5x cost, 4.5x decode latency (68 + 4.5×72 ≈ 392 ms, over budget), ~(312/69)² ≈ 20x attention compute. Worst-quality languages get worst latency + cost. Fixes: temperature-sample the vocab-fitting corpus, floor each language’s allocation, track fertility per language.

Measuring quality

• BLEU = geometric mean of clipped 1-4-gram precisions × brevity penalty. One zero precision zeros the score. Cost of an error is positional (how many 4-gram windows cover it), not semantic — it can rank a factual inversion (0.643) above a correct paraphrase (0.000). Not comparable across languages or tokenizations (use sacreBLEU signature); differences need paired bootstrap resampling.
• chrF2: character n-gram F-score, β=2 (recall-weighted), n=1-6, no zero cliff, no word tokenizer. Fixes morphology, not meaning.
• COMET / BLEURT: learned regressors, ~0.4-0.6 Kendall tau vs ~0.2 for BLEU; do separate meaning. Reference-free COMET-QE is the deployable serving-path model. Optimizing against them = reward hacking (keep decode metric ≠ reported metric).
• MQM: human error spans + severities, ~$12k/round, launch-blocking on a fixed set.
• Gate per-pair (drop ≤1.0 chrF / ≤0.02 COMET vs prod on a fixed 500-segment set), never aggregate: a 0.05% pair can go to garbage and move aggregate chrF by 0.003.
• Tail launches can’t be A/B tested: a 0.05% pair would need ~3.4 years to power. Use offline per-pair gates (blocker) + side-by-side preference panels + online A/B for high-traffic pairs + guardrail metrics.

Serving: two lanes, exact cache, guards

Request → normalize/segment/langID → EXACT cache (30% hit, 3ms)
   miss → Router
      ├─ Lane B  0.4B distilled   620 req/s/GPU   60% traffic   p50 140ms
      └─ Lane C  8B MoE            23 req/s/GPU    10% traffic   p50 1075ms
   → Guards → write cache → respond

• Decode dominates and is memory-bandwidth-bound: time/step = weight bytes read / bandwidth. Lane C is 20x bigger → 13x slower decode; batching doesn’t help a single sequence. Lane C serves document mode + low-resource only.
• Cache is exact-match on a version-keyed source: key = hash(nfkc_casefold(src), src_lang, tgt_lang, model_version, formality). model_version is load-bearing (else cache is a permanent record of every shipped bug). Never semantic cache (a one-numeral near-match returns confidently wrong).
• Document context helps ~2-4 chrF on a contrastive set but ~0.2 aggregate (only 5-10% of sentences are discourse-sensitive). Ragged lengths push padding waste >50%. Use document mode only; on interactive path carry a formality/glossary tag (one prefix token).

Cost: the router is the architecture

• Fleet ~322 GPUs, $15.5k/day ($5.6M/yr, ~$12.88/M requests). Don’t forget the QE-guard row (runs on ~96% of misses, its own fleet).
• Lane C is 10% of traffic but ~82% of the decode fleet (227 of 278 GPUs), because 23 req/s vs 620.

• Routing threshold is ~5.4x the cache lever. Caching is a latency feature; the router is the cost architecture. Beam-width is the biggest number but the first to be wrong (cuts quality exactly in the low-resource lane). List price ~373x marginal cost — the tail sets the price, not the request in front of you.

Failure modes

• Hallucination (target-side loss again): when the encoder gives no signal, the decoder emits its distribution mode (often religious text — the only large corpus for many languages). Guards cheapest-first: H01 length ratio (free), H02 cross-attention mass <0.25 (free), H03 COMET-QE below per-pair floor (catches the subtle dropped negation nothing else does).
• Gender bias: source lacks a distinction the target forces (Turkish o). Not fixable in-model; detect ambiguity (swap-pronoun score) and emit both with a note.
• Numeral/entity mangling: harness code, not model. Compare anchors not sets, split on non-digits, parse dates per locale, canonicalize notation, accept spelled-out numbers; repair via constrained decoding (force source numerals). A naive digit-set guard is worse than none.
• Degenerate repetition: created by beam search’s length-normalized score, not the model. Guards: n-gram blocking, coverage penalty, length-ratio check.
• Rejected: pivot through English (deletes T-V distinction, evidentiality — a pivot deletes categories, not just corrupts), 70B LLM on interactive path (20-50x cost; use as distillation teacher only), semantic cache, aggregate chrF gate, human review of all low-confidence output (24M reviews/day).