How to design a video recommender

In this lesson, we’ll build a video recommender: the system that picks the roughly 20 videos a user sees next, out of hundreds of millions, in the tens of milliseconds between opening the app and the home feed appearing. We’ll derive the design instead of describing it, so by the end you’ll be able to say why each part has to be the way it is and defend the whole shape in an interview.

Three ideas run through the lesson:

1. The two-stage funnel is forced, not chosen. Simple arithmetic rules out every one-stage design.
2. A single engagement objective can be gamed three different ways. Each way needs its own defence.
3. The training data comes from the system’s own past decisions. That corrupts both the model and its evaluation unless you spend traffic to prevent it.

Types of recommender

Before we design anything, let’s line up the four approaches you’ll hear about and what each one trades away. We’ll commit to one by the end of this section, and knowing why the other three lose is half of what an interviewer is checking for.

• Content-based filtering: recommends items whose features (topic, text, thumbnail) resemble items the user already liked. Tradeoff: it rarely surprises anyone, because it can only suggest more of what the user has already seen.
• Collaborative filtering: recommends what similar users liked (“people like you watched X”). Tradeoff: it needs interaction history, so it fails on brand-new users and brand-new items (the cold-start problem).
• Matrix factorization: an efficient collaborative-filtering method that learns one vector per user and one per item directly from the table of past interactions. Tradeoff: a brand-new item has no interactions, so its vector never gets learned.
• Two-tower / embedding retrieval: two networks, one reading the user and one reading the item, each producing an embedding; their similarity is a single dot product. Because the item embedding is built from item content, a brand-new item is usable immediately, and all item embeddings can be precomputed for fast search. Tradeoff: the two sides only meet at that final dot product, which caps how much user-item interaction the model can capture.

flowchart TD
    R["Recommender approaches"]
    R --> CB["Content-based<br/>match item features to what you liked<br/>weak at discovery"]
    R --> CF["Collaborative filtering<br/>'people like you liked X'<br/>needs interaction history"]
    CF --> MF["Matrix factorization<br/>one learned vector per user and item<br/>new items never get a vector"]
    CF --> TT["Two-tower / embedding retrieval<br/>item embedding built from content<br/>handles new items, capped interaction"]
    style TT fill:#2d6a4f,color:#fff

We’ll use two-tower retrieval to propose candidates (highlighted above), then a separate ranker to order them. Everything below builds exactly that and shows why each simpler option loses. To build it, we first pin down what one request carries and what it has to return.

What goes in, and what comes out

The input is one request: a user id, that user’s watch history, their device, the time, their locale, and how many slots the surface has to fill.

The output is a slate, an ordered list of about 20 videos filling those slots. Each video shown is one impression, and the impression is the unit everything downstream is counted in.

In between, four models run in sequence:

The whole system, offline and online

The system runs on two clocks. Heavy work happens offline in batch: building the item index, retraining. The online path is what runs per request under a tight latency budget. Every slate we serve becomes tomorrow’s training data, which is the loop at the bottom.

flowchart TD
    subgraph OFFLINE["Offline (batch, nightly / daily)"]
      ITEMS["800M item content"] --> ITOWER["Item tower"]
      ITOWER --> INDEX[("ANN index<br/>precomputed item embeddings")]
      USERS["500M user histories"] --> UTOWER["User tower<br/>recomputed nightly"]
      LOGS[("Yesterday's impressions<br/>+ delayed labels")] --> TRAIN["Retrain towers + ranker<br/>debiased: IPW, logQ"]
    end
    subgraph ONLINE["Online (per request, ~75 ms)"]
      REQ(["Request"]) --> RT["Real-time state<br/>mean of last 20 watches"]
      RT --> RETR["Retrieval: ANN + 5 more sources<br/>-> ~2,000 candidates"]
      RETR --> PRE["Pre-rank -> 300"]
      PRE --> RANK["Ranker: 8 heads over 300"]
      RANK --> VAL["Value model -> 1 score"]
      VAL --> SLATE(["Slate: 20 impressions"])
    end
    INDEX --> RETR
    UTOWER --> RETR
    SLATE -.logged.-> LOGS
    TRAIN -.new weights.-> ITOWER
    TRAIN -.new weights.-> RANK

Words this article uses

• Embedding: a fixed-length list of numbers standing in for a user or item, learned so that similar things land near each other.
• FLOP: one floating-point operation; the standard way to price arithmetic. MFLOP, GFLOP, TFLOP, PFLOP and ZFLOP are a million, billion, trillion, quadrillion and sextillion of them.
• QPS: queries per second. DAU: daily active users.
• A100: a specific data-centre GPU, used as the unit of compute you can rent.
• MLP (multi-layer perceptron), the plainest neural network: a stack of matrix multiplications with a nonlinearity between them.
• Head: a small output layer bolted onto a shared network body, one per thing you want to predict.
• ANN index (approximate nearest neighbour): a data structure that returns the closest few vectors out of hundreds of millions without comparing against all of them, at a cost that grows roughly with the log of the corpus, not linearly.

Concepts you’ll need (quick recap)

These come up below and are worth one line each so this page stands alone.

• Bandit: an algorithm for repeatedly choosing between options whose payoffs you only learn by trying them, balancing exploiting what already looks good against exploring what might be better. Thompson sampling does this by drawing a plausible payoff from each option and picking the winner; UCB (upper confidence bound) ranks options by an optimistic estimate. If you want the mechanics, see the reinforcement-learning chapter (optional).
• Off-policy evaluation: estimating how well a new policy would have done using only logs generated by the old one.

Why there have to be two stages

One calculation decides the whole architecture: what it would cost to score every video for every request. We can work it out directly, starting with the dimensions:

• Corpus: 800 M eligible videos, +500 k/day.
• Traffic: 500 M DAU × 40 requests/day = 20 billion requests/day ≈ 231 k QPS mean, ~509 k QPS at peak (traffic is not spread evenly, so the fleet is sized for the busy hour).
• Budget: 100 ms end to end, ~80 ms for ML.

Now price the simplest design we could ship: run the good ranking model on every video and sort. A serious ranker is about 4.4 MFLOP per (user, item) pair. So:

• Scoring all 800 M items for one request ≈ 3.5 PFLOP, which is about 23 seconds on one A100 against an 80 ms budget.
• At peak that is ~12 million A100s. Twelve million GPUs to answer one product’s requests.

A request can afford to score about 300 items, so the gap is log10(800M / 300) ≈ 6.4 orders of magnitude. No compression, quantization, or batching trick closes six orders of magnitude; those buy 2x to 10x. The one structural fix is to stop scoring most of the corpus: replace the exhaustive scan with an operation whose cost does not grow with corpus size, a precomputed index plus an ANN lookup.

That forces a funnel of stages, each spending more per item because fewer items remain:

flowchart TD
    C(["Corpus · 800,000,000"]) --> X["Score everything with the ranker<br/>3.5 PFLOP/request · 23 s each<br/>~12,000,000 A100 at peak"]
    X --> NO(["IMPOSSIBLE<br/>a gap of 6.4 orders of magnitude"])

    C --> R["Retrieval · 6 sources<br/>ANN + precomputed item embeddings<br/>cost independent of corpus size"]
    R --> P2(["2,000 candidates"])
    P2 --> PR["Pre-rank · 0.05 MFLOP/item<br/>tiny model, no new feature fetch"]
    PR --> P3(["300 candidates"])
    P3 --> RK["Full rank · 4.4 MFLOP/item<br/>the expensive model"]
    RK --> P4(["300 scored"])
    P4 --> SL["Slate re-rank<br/>diversity · freshness · integrity demotion"]
    SL --> P5(["20 slots"])

    style X fill:#9d0208,color:#fff
    style RK fill:#2d6a4f,color:#fff

Two things to take away. First, the arithmetic tells you which design is impossible, not what the possible one costs. Once the work is cut to 300 items, FLOPs stop being the binding constraint; the real fleet is sized by memory and feature fetch and lands one to two orders of magnitude above that ~4.5-A100 floor (worked out in the cost section).

Second, the pre-ranker exists for the same reason one level down: a cheap model guarding an expensive one. It is about 88x cheaper per item and cuts the ranker’s input from 2,000 to 300, so it pays for itself roughly 75 times over. The rule for any guard stage: it may be a smaller model, but it must run only over features you already have in hand, so it adds no new feature fetch.

This framing holds as long as three things are true, and each has a regime where it fails:

1. The item side of retrieval can be computed without knowing the user (what makes precomputation possible at all).
2. The corpus is large enough that scoring it exhaustively is hopeless. At 100 k items, one stage is correct and all of this is over-engineering.
3. Latency is bounded. You can add machines to serve more requests, but you cannot add machines to make one request’s sequential work finish sooner.

The first of those assumptions is the one to test next: can the item side really be built without knowing the user? That is exactly what candidate generation has to deliver.

Candidate generation

The first model turns 800 million videos into ~2,000 plausible ones. Almost everything about its design follows from one requirement: 500 k new uploads a day have to be retrievable before anyone has watched them.

Why matrix factorization fails here

Matrix factorization treats history as a giant table R of users by items and factors it into two thin matrices, R ≈ U Vᵀ: one learned vector per user and per item, fit by minimizing error over observed interactions.

For a brand-new item, the loss’s gradient with respect to that item’s vector is a sum over that item’s interactions. There are none, so the gradient is exactly zero, so the vector stays at its random initialization forever. This is not sparsity you can regularize away; it is definitional. At 500 k new videos a day, a retrieval model that cannot represent a new item has forfeited its entire supply side.

Two-tower retrieval

The fix is to make the item’s representation a function of what the item is, not of who has watched it:

score(u, i) = < f(u; user features), g(i; item content features) >

Two separate networks, their outputs compared by one dot product. This buys two things people often run together:

• g reads item features, not an item id: title text, thumbnail, channel, category, duration, language, upload time. A video uploaded 90 seconds ago has all of those, so its embedding is meaningful at zero interactions. This is the real reason to prefer two-tower over matrix factorization.
• g depends only on i, so it can be precomputed. All 800 M item embeddings (800 M × 128-d fp16 ≈ 205 GB) are built once per index cycle and loaded into an ANN index. Per request: one forward pass of f(u), then one ANN query.

The price is a cap on interaction, and the usual telling gets it wrong. A dot product is a sum of user-feature × item-feature products, so it does express interactions: “short videos on cellular, long videos on wifi” is one such pattern (it is rank 1, and two dimensions carry it exactly). What is capped is the number of independent interaction patterns, to at most d. A random 500×500 interaction table keeps only about 63% of its structure in its top 128 directions, so the remaining ~37% cannot be expressed at retrieval and has to be recovered later by the ranker. You buy sublinear retrieval by capping interaction at rank d, and buy the rest back in the ranker. That, in one sentence, is why there are two stages.

Why d = 128? An ablation settles it. Recall@2,000 (the share of held-out satisfied watches that appear anywhere in the 2,000 candidates) climbs with d, then stops paying:

Between 128 and 256, memory doubles and recall moves 0.005. So 128 is the last doubling that pays, for this corpus. A smaller catalog wants a smaller d; the number is a decision, not a default.

Training the towers: sampled softmax and the logQ correction

A positive is a (user, item) pair whose watch cleared a satisfied-watch threshold (not a click; training retrieval on clicks would import clickbait into the candidate pool, where no later stage can remove it). Negatives come mostly from the other items in the same training batch, which is what makes the loss affordable: you get B − 1 negatives for free from items already loaded. Always split temporally (train days 1–28, validate day 29, test day 30); a random split lets tomorrow’s data leak into today’s training.

The ideal loss is a softmax over the whole corpus, but its denominator sums over all 800 M items per example, which nobody can compute. Summing over the in-batch sample instead is cheaper but biased: popular items appear as negatives far more often than they should, so they get pushed down too hard. The standard repair is importance sampling: divide each sampled term by its sampling probability Q(j), which inside the softmax is the same as subtracting log Q(j) from the score. That subtraction is the whole correction:

corrected logit(u, j) = < f(u), g(j) > - log Q(j)

Skip it and the model converges to s(u,i) − log(popularity) instead of s(u,i). For a head item that is about a 6.9-nat handicap, a factor of ~1,000 in softmax probability at an identical dot product: an unchosen, untunable popularity penalty. Some teams want a popularity penalty; the point is to choose it deliberately, not ship it by accident. Two things to get right: apply the subtraction inside training only (applying it at serving re-introduces exactly the boost you removed), and estimate Q from a decayed counter with a sensible half-life (a one-day half-life against a daily index cycle is typical).

Six sources, not one

The two-tower model is one retrieval source, not the retrieval system. A serious feed runs six, and the per-source quota (how many candidates each may contribute) is where the diversity policy actually lives.

Every source has a different failure mode, so running them together means no single source’s bias becomes the system’s bias. Each is also independently debuggable, A/B-able, and killable. And the quota is the only place diversity can be set: a diversity penalty at re-ranking can only reorder what retrieval already returned, so if co-watch supplied 90% of the pool, no re-ranker can fix it.

One ordering detail matters: dedupe before you truncate. Each source over-fetches, then the union is deduped, per-channel capped, and only then cut to quota. Truncating first silently loses candidates that appeared in two sources (about a 7% leak across all six), and it loses them non-uniformly, shifting the pool’s composition. Dedupe first and the quotas actually add up.

Ranking

The expensive model scores the 300 survivors and decides the slate. What it takes as input, why it predicts eight things, and how eight predictions become one number all follow from one premise: any objective you can write down will be gamed.

What the 1,500 input dimensions are

The ranker reads seven feature families, ~1,500 dimensions total. The two columns worth reading are Refresh (how often the values change, which decides what can be precomputed) and Trap (how each family goes wrong):

Two consequences follow. This is where the interaction rank retrieval gave up comes back: connection type and duration sit in the same concatenation, feeding a shared trunk that is nothing but interaction terms, with no rank ceiling. The item families are also what cost you at serving: about 1,936 bytes × 300 candidates ≈ 0.58 MB of scattered reads per request, which sizes the fleet far more than the arithmetic does. The 532 user-side dimensions are fetched once per request, not per candidate, which is why widening the user side is nearly free and widening the item side is not.

Why eight heads, not one

A single objective is a specification, and every specification is gameable. Predict clicks and you get thumbnails that lie. Predict watch time and you get the watch-time trap (next section). Predict likes and you get “smash that like button.” The defence is a set of heads chosen so no one lever moves all of them the same way.

The cheapest large quality win in most recommenders is adding a negative head. The positive heads all correlate with each other, so each adds little new information; the negative signal is nearly independent of them. Measured here, adding the not-interested head moved offline AUC by only 0.002 but moved 28-day retention by 0.9 points, and that gap between a tiny offline move and a large online one is the whole argument.

Predicting eight things from one shared trunk creates a problem: tasks whose gradients point in opposite directions destroy each other’s representation (clickbait maximizes clicks and minimizes satisfaction). MMoE (multi-gate mixture of experts) replaces the single trunk with several small trunks (“experts”) and gives each task a learned gate deciding how much of each expert it reads, so conflicting tasks route to different experts. Trained both ways here, click AUC was unchanged (click was already winning the fight), but satisfaction rose 4.9 points and share 3.5 points, the two tasks that had been losing, for 18% more parameters.

Combining eight predictions into one score

The score is a weighted sum, score = Σ w_k · φ_k(p_k). Four details decide whether it works:

1. Transform before weighting. Base rates span 0.58 to 0.00006, four orders of magnitude. A linear blend of raw probabilities cannot let the share head matter at any sane weight. Use log space: weighting log-probabilities is weighting each head’s relative lift over its base rate, which is what you actually mean.
2. Watch time needs its own transform. Raw seconds has a heavy tail (one 3-hour stream swamps a batch). The head predicts watch fraction, and a separate saturating duration term (log1p(minutes)) restores a mild preference for longer content without letting length win on arithmetic alone.
3. Weights come from a value model. Eight weights can’t be grid-searched (four values each is 65,536 variants). The honest source is an out-of-band target (200 k surveys plus 28-day retention plus reported regret) with weights fit so the score predicts it. Because the target is collected outside the recommendation loop and resolves 28 days later, the ranker cannot game it within a session. It is refit quarterly.
4. Integrity is added, not multiplied. The combined score has no fixed sign (it’s about −24 on a weak item and +8 on an ordinary one). Multiplying by an integrity score in (0, 1] therefore promotes weak items (−24 × 0.5 = −12, which is higher) while demoting ordinary ones, a non-monotone bug that runs exactly backwards. The demotion has to be score + log(integrity), which is monotone decreasing for every item. Adding a log is multiplication in the space the score is a log of, so this is the multiplier people meant.

The payoff is visible in one slate. A clickbait clip with more than twice the click probability of an honest tutorial finishes last, because its not-interested (4.5x its base rate) and report (3.5x) predictions enter at weights of −2 and −4:

That is the watch-time trap’s threat model producing an ordering instead of just describing one. (The livestream’s 3 hours buy it a duration bonus of only ~2.1 points, nowhere near enough to overcome a 0.06 watch fraction, which is exactly what “saturating” was for.)

Labels arrive on different clocks

Every head is trained on the same logged impression rows, but each head’s label arrives on its own schedule, and a fixed 24-hour training window turns “hasn’t happened yet” into “didn’t happen”:

A censored positive is not a missing row: it is a row labelled negative. Worse, the censoring rate depends on the item: a subscribe after a 90-second clip lands in-session, one after a 25-minute video usually does not (9% censored for short videos, 47% for long ones). At the subscribe weight, that hands short videos about 1.62 score points, larger than, and opposite to, the 0.94-point duration term you deliberately chose. The accidental prior beats the deliberate one, and it is the watch-time trap re-entering through the data pipeline.

The fix is per-head label maturity windows: hold each head’s rows until that head’s window closes (train click/watch/like on T−1, share/subscribe on T−3, report on T−7). Two tempting alternatives are both wrong: one global 7-day window costs the click head a week of staleness for no reason, and dropping unmatured rows reproduces the bias exactly, because the rows you drop are the long-video rows. Only holding is unbiased. And train on features logged as served, so training and serving are the same code path by construction.

The watch-time trap

The ranking design above looks the way it does because of watch time, the metric every recommender is asked to maximize. It fails in three separate ways.

1. The product decomposes, and one factor is cheap to game. E[watch time] = P(click) × E[watch | click]. An honest item at 0.040 click and 180 s comes out to 7.2 s; a clickbait item at 0.140 click and 62 s comes to 8.68 s. B wins on total watch time while being worse on every per-viewer measure, because inflating clicks costs a thumbnail while raising retained attention is genuinely hard.

2. Total watch time is a duration prior in disguise. A 3-hour livestream watched 6% is 648 s and beats a 6-minute tutorial watched 90% (324 s), so creators pad. Ranking on watch fraction instead inverts the bias: a 15-second clip watched 100% wins everything. You need both, a fraction head plus a separate saturating duration term.

3. The sign flips across timescales. Within a session, watch time and satisfaction correlate positively. Across sessions they correlate negatively:

This is the dangerous one, because every short A/B endorses the climb: a two-week experiment cannot see D28. The design answers in three parts, each of which appears elsewhere in this article: explicit satisfaction terms in the score, a value model whose target resolves 28 days out, and a permanent long-term holdout (a slice of users, here 0.5%, kept on a frozen policy and never reassigned). Without the holdout, the sign flip is undetectable, and the system optimizes itself into the corner one confidently-shipped A/B at a time.

Position bias and the feedback loop

The system’s training data is produced by the system itself. That single fact is why training on raw clicks makes a model of your previous model.

A click is not pure evidence of relevance, because the user has to look at a slot before they can click it:

P(click | item r at position j) = examination(j) × relevance(r)

Examination is the probability the user looked at position j at all. We can measure it honestly by randomly swapping slot contents on a small traffic slice (0.5% here), so that what sits in a slot is unrelated to how good it is:

Position 1 is looked at nine times as often as position 20. Training on raw clicks fits the product, and since position was assigned by the old model, the new model just learns “items the old model liked are good.” Iterate that and you get a system that is very confident about a shrinking set of items.

Inverse propensity weighting (IPW) fixes it by weighting the label by the inverse of its examination probability. The subtle part is where the weight goes: the corrected label is click / examination(j), so the weight rides on the click, not on the whole row. Put 1/e on every logged row instead and it becomes a constant factor on that position’s entire objective; it cancels, and you converge to the biased quantity exactly as if you’d done nothing. With it on the click, position cancels in expectation and the fit recovers relevance:

The middle column is the common bug: identical to no correction at all.

The reflex next is to clip large weights, but price it before importing it. On a 20-slot slate, examination bottoms out at 0.09, so weights span only 1.23 to 11.1; there is no tail to trim. Clipping either never fires or costs real bias for negligible variance reduction. Use instead a propensity floor at the slate’s own minimum (0.09), dropping events below it, which are logs from another surface or corrupted rows, plus self-normalized IPS (dividing by the sum of weights, not the count). Also interpolate examination across all 20 positions before using it; the swap only measures seven, and dropping the unmeasured thirteen would itself be a position-dependent bias.

Finally, pay for the randomized swap instead of harvesting natural position variation, which is confounded (an item was shown low because the model scored it low).

The corpus you can actually learn about

Position bias is one half of the loop; the other half is that most of the corpus is never shown:

800 M eligible videos
videos with >= 100 impressions/day        4.1 M   = 0.51 %
videos with >= 1,800 impressions/day      0.9 M   = 0.11 %

Ninety-nine and a half percent of the corpus generates no training signal, so the “800 M corpus” is really a ~4 M corpus with a very expensive index attached. An item outside that set never gets impressions, never gets data, and never improves: a closed loop with no way in.

Exploration is the only way in. Reserve 1 in 20 slots for items the model does not yet rate, ranked with a Thompson/UCB bonus. How much is needed can be derived: a new video needs about 1,827 impressions before its CTR estimate is trustworthy to 20% relative, so graduating the daily upload cohort costs 500 k × 1,827 ≈ 914 M impressions, which is 0.23% of the 400 billion impressions served daily. So the binding question is not “can we afford exploration” but “are we spending it on the right items”: the fresh pool and wide-posterior items, not uniformly. (Log the realized exploration share per request, or an A/B result is uninterpretable.)

The reason teams kill exploration is that its two instruments disagree:

2-week A/B (exploration on vs off):   engagement -0.4 %   -> "kill it"
90-day persistent holdout:            coverage 0.51% -> 2.8%
                                      creator D30 retention +11 %
                                      engagement +1.2 %

Inside a two-week window exploration is only ever a cost, because its benefit is a supply-side effect that takes months. The persistent holdout is the only instrument that can see it, which makes it an infrastructure decision instead of an experiment.

Cold start

Cold start is serving something the system has no history for, and it comes in three flavours with genuinely different fixes.

New user. No history, so the long-term tower has nothing to encode. Only context is available: locale, device, time, referral, and whatever onboarding collected. The measurable result is that asking beats modelling:

new-user session-2 return rate
    popularity-by-locale only                         0.42
    + 3 explicit topic picks at onboarding            0.57
    + Thompson bandit over 40 topic clusters          0.61

Three onboarding taps are worth more (0.42 → 0.57) than any model you can build on top (0.57 → 0.61), because they are the only information that exists. The bandit works after ~15 interactions (one session) only because the 40 topic clusters share a hierarchical prior: nearby topics behave similarly, so a pull on one arm updates its neighbours. A flat bandit over 40 independent arms would need closer to 600 interactions.

New item. Half-solved by construction: the content tower gives a usable embedding at upload, so a 90-second-old video is retrievable. Not solved: it gives no reliable engagement estimate, which is the 1,827-impression number. So a fresh pool gives every new item a guaranteed impression budget, spread across diverse cohorts (all shown to one interest cluster would estimate that cluster’s taste, not the item’s), and graduates it when its CTR posterior is tight enough or at 72 hours.

Stale user vector. A user who just watched six pasta videos should see cooking next; if the tower is recomputed nightly, they won’t. The reflex is to recompute the tower per request, but that is the wrong fix, and “too expensive” is the wrong objection (the forward pass is ~0.001 A100). The real reasons are that it costs ~6 ms of history-read latency per request, and that the benefit is near zero anyway: the long-term tower is an average over months, so six new videos against a 2,000-event history move it by ~0.3%. The correct decomposition is to split the representation by refresh cost:

The expensive representation updates slowly; the cheap one (a plain average of embeddings already in cache) updates instantly and feeds both retrieval and the ranker. That is the answer to “your recommendations don’t react to what I just watched.”

Metrics, and why offline and online disagree

“How do you know it works” has an offline answer and an online answer, and in this system they disagree in sign about three times in five.

Offline, four measurements, each with a trap:

Calibration is not optional here. In a single classifier you can miscalibrate and fix it with a threshold. Here eight heads are multiplied by weights and summed, so an overconfident head silently rescales its own weight.

Online, the primary metric is satisfied-watch per session; the guardrails are D7/D28 return rate (the only metric that sees the watch-time sign flip), survey satisfaction, not-interested/report rates, corpus coverage and impression Gini (how unequally impressions are spread, 0 even to 1 winner-take-all), and topic entropy of slates and of user histories (how narrow a diet is).

The two disagree, and it is structural, not statistical. Across five real launches, three showed opposite signs offline vs online, and twice the offline winner was the online loser. Three mechanisms:

1. Offline data is logged by the old policy. A new model’s favourite items were rarely shown, so they have no labels; the evaluator drops them or scores them as negatives, and either way the model that ranks most like production scores best. Offline AUC rewards resembling the logging policy, exactly the property a candidate model is trying not to have.
2. Offline is per-request; the outcome is per-session. Crowding-out and repetition are invisible in per-request AUC.
3. The correction applied at serving may not be applied in eval. Train IPW-weighted but evaluate on raw clicks and you carry back the bias you removed.

Three fixes, in order of value: a permanent 0.1% uniform-random slice (20 M requests/day serving a random sample from the candidate pool, the only data in the system that is not a function of the model being evaluated), doubly-robust off-policy estimation on it, and always logging the candidate set and propensity (retrofitting them is impossible; a log without them is permanently un-debiasable).

Failure modes

Six ways this system degrades in production. Four are feedback loops, where the output becomes the input, and the useful habit is to read each as a gain: if one pass multiplies some concentration by a factor above 1, the failure is a certainty on a schedule, not a risk.

Filter bubbles. The ranker prefers items like past engagement; those get shown; history narrows; the estimate narrows. Measured, 30-day topic entropy falls from 3.1 to 1.9 nats over twelve weeks unmitigated. Per-source quotas recover most of it (to 2.6), MMR slate re-ranking a little more (to 2.9). The quota does the heavy lifting because a re-ranker can only reorder what retrieval returned. Report entropy of user histories, not just of slates: a diverse slate the user never clicks still leaves history narrowing.

Popularity amplification. Popularity is both a feature and an outcome, so showing a popular item makes it more popular. Impression-share Gini climbs from 0.71 to 0.83 over six months unmitigated. The main fix is residualizing popularity features: feed views − E[views | age, category, channel size] instead of raw counts, so the model learns “this beat its cohort” (new information) instead of “popular things get clicked” (a fact about the past that closes the loop).

Offline eval optimism. Your eval set is sampled by your current model, so it flatters models that resemble it, and the flattery grows every time you ship. The uniform-random slice is the only exit.

Clickbait / engagement bait. The watch-time mechanism from the supply side: creators A/B their thumbnails against your ranker at a scale you can’t match. Held by the satisfaction head carrying real weight, the not-interested head, a title/thumbnail-to-content match feature, and post-click retention as a slow-decaying quality prior.

Training/serving skew. A feature computed one way in training and another at serving. The classic outage: user_watch_count_7d from a complete batch table in training but from a streaming store with a 40-minute lag at serving, so the served value is systematically low for exactly the active users in every request. Offline AUC 0.79, online engagement −2.1%. The skew is correlated with what you’re predicting, which is what makes it lethal. Fix: log features as served and train on those logs.

Integrity coupling. Borderline content (close to the policy line but not over it) is reliably engaging, so a ranker with no integrity input will promote it. The integrity score belongs in two places: as a ranker feature (which generalizes to items the classifier never scored) and as a graded demotion in re-rank (+ log(integrity), which is auditable and enforceable). A post-hoc filter is strictly weaker than both. If you want the safety side, the harmful-content classifier that produces this score is covered in the content-moderation chapter (optional).

Every failure reduces to a mechanism, a measurement, and a control:

Scale and cost

Pricing takes three passes, the arithmetic floor, the latency budget, and the fleet you actually rent, and the gap between the first and the third is the point. FLOPs tell you which design is impossible; bytes tell you what the possible one costs.

Arithmetic floor. Ranking 300 candidates is ~1.3 GFLOP/request → ~4.5 A100s at peak; the pre-ranker adds ~0.3; the floor is 4.8 A100s ($230/day, ~$1e-8/request). The user-tower refresh is genuinely free (~1 GPU-second nightly). This is a floor, not a bill.

Real fleet. Serving is memory- and fetch-bound once the candidate set is small, so the fleet is sized by how many bytes must be online and how many replicas survive a shard loss:

ANN index          ~410 GB (vectors + HNSW graph), 16 shards x 3 replicas   ~48 nodes
ranker item feats   800 M x 1,936 B = 1.55 TB, 20 shards x 3 replicas        ~60 nodes
                                                                            ----------
                                                                             ~108 nodes  (~22x the FLOP floor)

Those ranking nodes run at about 7% arithmetic utilization, because they were sized to hold 1.55 TB, not to do math. That is the whole “FLOP floor is not the bill” argument made concrete, and it is the design lever: shrinking bytes per candidate (int8 embeddings, hashing the channel-id tail, dropping the ASR embedding if it doesn’t earn its keep) moves the fleet; a cheaper ranker does not.

Latency. The ~80 ms ML budget, spent:

user feature fetch (KV)                       8 ms
user tower forward pass                        1 ms
6 retrieval sources, IN PARALLEL             14 ms   <- slowest source, not the sum
dedupe + per-channel cap + seen-filter        3 ms
pre-rank 2,000 -> 300                          5 ms
feature hydration for 300 candidates         24 ms
full ranker, 300 candidates                  16 ms
value combine + integrity + MMR               4 ms
                                            ------
                                             75 ms  ML path, p50

About 38 of the 75 ms is fetch, not arithmetic. Even the 16 ms of “ranker” is mostly not ranking: the dense math for 300 candidates is under 10 microseconds; the rest is embedding gathers and a batch too small to fill an A100. Batching the 300-candidate hydration into one multi-get per shard buys more than shaving the ranker would. (Note the convention: this 100 ms is measured server-side at the gateway and excludes the client round trip.)

Training data. 400 billion impressions/day × ~600 B ≈ 240 TB/day raw. Negatives are downsampled 20:1 to ~26 TB/day. Downsampling breaks calibration (the class mix goes from 1:15 to about 56% positive), which matters because the heads combine arithmetically, so correct the logits analytically for the sampling rate instead of re-calibrating empirically.

Alternatives considered and rejected

The middle column is the one that matters most: if an alternative isn’t tempting, its rejection isn’t worth studying.

Where an LLM does earn its cost is upstream and offline: generating content-understanding features for the item tower (topic, style, thumbnail-to-content consistency), computed once per video at upload instead of per user per request, 500 k inferences a day instead of trillions, put exactly where cold start needs it.

Final design

flowchart TD
    REQ["Request: user, history, device, time, N slots"]
    subgraph RET["Candidate generation - 6 sources, quota-capped to ~2,000"]
      TT["Two-tower retrieval<br/>ANN over 800M"]
      OTH["Subscriptions · trending · fresh · social · exploration"]
    end
    REQ --> RET
    RET --> POOL["Pool ~2,000<br/>dedupe · per-source quota"]
    POOL --> PR["Pre-ranker<br/>tiny model, 2,000 to 300"]
    PR --> RANK["Ranker<br/>MMoE, 8 task heads over 300"]
    RANK --> VM["Value model<br/>8 predictions to 1 score"]
    VM --> RR["Slate re-rank<br/>diversity · freshness · integrity demotions"]
    RR --> SLATE["Slate: 20 impressions"]
    subgraph OFF["Offline, daily"]
      LOG["Logged impressions + delayed labels<br/>per-head maturity windows"]
      IPW["IPW / randomized-slot debiasing"]
      TRAIN["Retrain towers (sampled softmax + logQ)<br/>and ranker heads"]
    end
    SLATE -.impressions.-> LOG
    LOG --> IPW --> TRAIN
    TRAIN -.new weights.-> TT
    TRAIN -.new weights.-> RANK

Left to right is one request on a ~75 ms budget: retrieval never touches corpus size, the pre-ranker cheaply cuts 2,000 to 300, the ranker spends its FLOPs on 300, and the value model collapses eight predictions into the one number the slate is sorted by. The dashed loop is what makes the system hard: every slate it serves becomes tomorrow’s training data, so debiasing and a randomized exploration slice are load-bearing, not optional.

Conclusion

• The two stages are forced, not chosen. A request can score ~300 of 800 M items; the 6.4-order-of-magnitude gap rules out every one-stage design.
• A single engagement objective is gameable three ways (product decomposition, a hidden duration prior, and a session sign-flip), so the ranker predicts eight outcomes and a value model combines them.
• The system trains on its own past decisions. Position bias, popularity amplification, and label-window censoring all trace to that loop; inverse propensity weighting, per-head maturity windows, and a uniform-random exploration slice are the fixes.
• Exploration looks like a cost and is an investment. Its benefit is invisible to any two-week A/B and shows only in long-horizon holdouts.
• FLOPs tell you which stage is impossible, not what the possible one costs. Serving is bound by embedding memory, feature fetch, and tail latency, not arithmetic.

One line to remember: retrieval buys sublinear cost by capping interaction at rank d and the ranker buys the rest back, and every other decision in this system defends an objective the data itself is trying to game.

Further reading

• Covington, Adams & Sargin, Deep Neural Networks for YouTube Recommendations (RecSys 2016): the canonical two-stage retrieval-and-ranking design.
• Yi et al., Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations (RecSys 2019): two-tower retrieval and the logQ correction.
• Ma et al., Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts (KDD 2018): the multi-task ranker.
• Joachims, Swaminathan & Schnabel, Unbiased Learning-to-Rank with Biased Feedback (WSDM 2017): inverse propensity weighting for position bias.
• Chen et al., Top-K Off-Policy Correction for a REINFORCE Recommender System (WSDM 2019): exploration and off-policy correction at scale.

Next: the event-recommender chapter.