How to design a video recommender

Pick ~20 videos out of 800M in ~75ms; a two-stage funnel is forced, a single engagement objective gets gamed, and the system trains on its own past decisions.

The funnel (forced by arithmetic)

• A serious ranker is ~4.4 MFLOP per (user, item). Scoring all 800M items ≈ 3.5 PFLOP/request ≈ 23s on one A100 vs an 80ms ML budget → ~12M A100s at peak.
• A request affords ~300 items; the gap is log10(800M/300) ≈ 6.4 orders of magnitude. No quantization/batching trick closes 6 orders (they buy 2–10x). Structural fix: stop scoring the corpus, use a precomputed index + ANN lookup (cost grows with log of corpus).

• Guard-stage rule: a cheaper model guarding an expensive one, running only over features already in hand (no new feature fetch). Pre-ranker is ~88x cheaper/item.
• Holds only while: item side is user-independent (precomputable), corpus is huge (at 100k items one stage is right), and latency is bounded (machines add throughput, not shorter sequential work).

Candidate generation: two-tower retrieval

• Cold-start: 500k new uploads/day must be retrievable before any watch. Matrix factorization fails: a new item’s gradient sums over its (zero) interactions → stays at random init forever, definitionally.
• score(u,i) = <f(u; user features), g(i; item content features)>. g reads item content (title, thumbnail, channel, duration, language, upload time), not an item id → meaningful at 0 interactions; depends only on i → precomputable.
• Precompute all 800M embeddings (128-d fp16 ≈ 205 GB) into an ANN index. Per request: one f(u) forward pass + one ANN query.
• Cost: interaction is capped at rank d. A dot product still expresses interactions but only ≤d independent patterns; the rest is recovered by the ranker. That is why there are two stages.
• d=128 by ablation: recall@2,000 is 0.71/0.79/0.83/0.835 at d=32/64/128/256. 128→256 doubles memory for +0.005; 128 is the last doubling that pays.

Training the towers

• Positive = watch clearing a satisfied-watch threshold (not a click; clicks import clickbait no later stage can remove). Negatives are in-batch (B−1 free per example). Always split temporally (train 1–28, val 29, test 30).
• In-batch negatives bias toward popular items → logQ correction (importance sampling): corrected logit = <f(u),g(j)> − log Q(j).
• Skip it → converges to s(u,i) − log(popularity): a ~6.9-nat (~1,000x softmax) popularity penalty, unchosen and untunable. Apply inside training only (at serving it re-adds the boost); estimate Q from a decayed counter (~1-day half-life).

Six retrieval sources, not one

• Quotas (of ~2,000): two-tower long-term 600, item-to-item co-watch 400, subscriptions 300, fresh pool (<48h) 250, trending in locale 250, social 200.
• Each source has a different failure mode → no single bias becomes the system’s; each is independently A/B-able and killable.
• Quota is the only home for diversity: a re-ranking penalty can only reorder what retrieval returned.
• Dedupe before truncate: over-fetch → union → dedupe → per-channel cap → cut to quota. Truncate-first leaks ~7% (dupes across sources), non-uniformly.

Ranking: eight heads + value model

• Ranker reads ~1,500 dims across 7 feature families. User side (532 dims) fetched once/request; item side (~1,936 B × 300 ≈ 0.58 MB scattered reads) sizes the fleet. Traps: store local hour (UTC = timezone bug); feed popularity residuals not raw counts; keep provenance to spot a source returning garbage.
• Why 8 heads: any single objective is gameable (clicks→lying thumbnails, watch time→trap, likes→“smash like”). Heads: P(click) 0.061, E[watch frac|click] 0.34, P(like) 0.011, P(share) 0.0008, P(subscribe) 0.0004, P(not-interested) 0.0021, P(report) 0.00006, P(survey≥4) 0.58.
• The cheap big win is a negative head: not-interested moved offline AUC +0.002 but 28-day retention +0.9 pts (positive heads correlate; the negative is near-independent).
• MMoE: conflicting tasks (clickbait maxes clicks, mins satisfaction) destroy a shared trunk; several experts + per-task gates route them apart. Here satisfaction +4.9 pts, share +3.5 pts for +18% params.

Combining into one score

• score = Σ w_k · φ_k(p_k). Base rates span 4 orders of magnitude.
• Transform before weighting: use log space (weight each head’s relative lift over base rate).
• Watch time: predict watch fraction + a separate saturating duration term (log1p(minutes)); raw seconds has a 3-hour-stream tail.
• Weights from a value model fit to an out-of-band target (200k surveys + 28-day retention + regret), refit quarterly; can’t be gamed in-session. (8 weights = 65,536 grid variants → not searchable.)
• Integrity is added, not multiplied: score + log(integrity) (monotone). Multiplying promotes weak negative-score items (−24 × 0.5 = −12, higher).

Label censoring

• Each head’s label arrives on its own clock; a fixed 24h window turns “not yet” into “didn’t”. Censored at 24h: click ~0%, share 22%, subscribe 31%, report 44%.
• Censoring depends on item: subscribe after short clip lands in-session (9% censored) vs long video (47%) → hands short videos ~1.62 score pts, beating the deliberate 0.94-pt duration term.
• Fix: per-head maturity windows (train click/watch/like on T−1, share/subscribe T−3, report T−7). One global 7-day window over-stales clicks; dropping unmatured rows reproduces the bias (dropped rows are the long-video rows). Only holding is unbiased. Train on features logged as served.

The watch-time trap (3 failures)

1. E[watch time] = P(click) × E[watch|click] — the cheap click factor is gamed (thumbnail is cheap, retained attention is hard).
2. Total watch time is a hidden duration prior (3h stream @6% beats 6-min tutorial @90%). Fraction alone inverts it (15s clip @100% wins) → need fraction + saturating duration.
3. Sign flips across timescales: within-session watch time correlates + with satisfaction, across-session −. (>90 min sessions: satisfaction 2.7 vs 4.1, D7 return 0.63 vs 0.71, regret 0.19 vs 0.04.) A 2-week A/B can’t see D28 → needs a permanent long-term holdout (0.5% on a frozen policy).

Position bias and the feedback loop

• P(click | item r at pos j) = examination(j) × relevance(r). Examination measured by randomized slot swaps on 0.5% traffic: pos 1=0.81 … pos 20=0.09 (9x gap).
• Training on raw clicks fits the product → new model learns “items the old model liked are good” → confident about a shrinking set.
• IPW: weight rides on the click, not the row: click / examination(j). 1/e on the whole row is a constant that cancels → identical to no correction (the common bug).
• No fat tail on 20 slots (weights 1.23–11.1) → clipping does nothing useful. Use a propensity floor at the slate min (0.09) + self-normalized IPS; interpolate examination across all 20 positions.

Corpus you can learn about

• Videos with ≥100 impressions/day = 4.1M (0.51%); ≥1,800/day = 0.9M (0.11%). ~99.5% generate no signal → an item outside gets no data, never improves.
• Exploration is the only way in: reserve 1 in 20 slots, ranked with a Thompson/UCB bonus, spent on fresh-pool and wide-posterior items (not uniformly). A new video needs ~1,827 impressions for a 20%-relative CTR estimate; graduating the daily cohort = 500k × 1,827 ≈ 914M = 0.23% of 400B daily impressions.
• 2-week A/B says −0.4% (kill it); 90-day holdout: coverage 0.51%→2.8%, creator D30 +11%, engagement +1.2%. Its benefit is supply-side, taking months → infrastructure decision, not an experiment.

Cold start (three flavours)

• New user: only context (locale, device, time, onboarding). Asking beats modelling: session-2 return 0.42 (popularity) → 0.57 (+3 topic picks) → 0.61 (+Thompson bandit over 40 clusters). Bandit works after ~15 interactions only via a hierarchical prior (a flat bandit needs ~600).
• New item: half-solved — content tower gives a usable embedding at upload; not solved — no engagement estimate (the 1,827 number) → fresh pool with an impression floor across diverse cohorts, graduate at tight CTR posterior or 72h.
• Stale user vector: nightly tower misses this session. Don’t recompute per request (not a FLOP problem: ~0.001 A100, but ~6ms history-read for a ~0.3% move on a months-long average). Split by refresh: nightly tower (durable taste) + last-20 mean pool (session intent, 0 FLOPs, already in cache) + per-request context.

Metrics: offline vs online

• Offline: AUC per head, NDCG@20, IPS/doubly-robust, calibration. Calibration is mandatory — 8 heads combine arithmetically, so an overconfident head silently rescales its weight.
• Online: primary = satisfied-watch/session; guardrails = D7/D28 return (sees the sign flip), survey satisfaction, not-interested/report, coverage, impression Gini, topic entropy.
• They disagree in sign ~3 of 5 launches, structurally: (1) offline data logged by old policy rewards resembling production; (2) offline is per-request, outcome is per-session; (3) serving corrections not mirrored in eval.
• Fixes (by value): a permanent 0.1% uniform-random slice (only data not a function of the model), doubly-robust off-policy on it, always log candidate set + propensity (un-retrofittable).

Failure modes (read each loop as a gain)

Scale and cost (FLOPs ≠ the bill)

• Arithmetic floor: ranking 300 ≈ 1.3 GFLOP → ~4.5 A100 at peak, +0.3 pre-rank ≈ 4.8 A100 ($230/day).
• Real fleet: serving is memory/fetch-bound → ANN index ~410 GB and ranker item feats 800M×1,936B = 1.55 TB, each ×3 replicas ≈ ~108 nodes (~22x the FLOP floor), ~7% arithmetic utilization. Design lever = shrink bytes/candidate (int8, hash channel tail), not a cheaper ranker.
• Latency (~75ms p50): ~38ms is fetch. Retrieval sources run in parallel (slowest, not sum); feature hydration for 300 = 24ms. Batch the multi-get before shaving the ranker.
• Training data: 400B impressions/day × ~600B ≈ 240 TB/day, negatives downsampled 20:1 → ~26 TB. Downsampling breaks calibration → correct logits analytically for the sampling rate.

One line

Retrieval buys sublinear cost by capping interaction at rank d; the ranker buys the rest back — and every other decision defends an objective the data itself is trying to game.