How to design personalized feed ranking

In this lesson, we’ll design a social network’s ranked feed end to end: what a post’s score should contain, where each term comes from, how a post travels from one author’s write into billions of candidate sets, and which of the resulting failures are bugs and which are properties of the system. By the end you’ll be able to write the objective, defend every term in it, and name the failure each term is holding back.

Three results carry the design. Maximizing engagement is a specific, knowable mistake, not a safe default. The weights in a multi-objective score cannot be read as written unless you divide by base rates first. And the most important effect of a feed ranker is invisible to the experiment that ships it.

The one-line shape: a viewer’s identity and session context go in, an ordered list of about 25 posts comes out, drawn from roughly 3,000 candidates that the viewer’s social graph and a set of recommendation indexes produced.

What makes a feed different from a catalog

The standard two-stage recipe (retrieve a pool of candidates, rank them) is correct but not sufficient. A cheap model narrows a large pool to a few thousand and an expensive model orders those; the video-recommender chapter derives it in full. Two ideas from there we’ll reuse constantly: that two-stage retrieval pattern, and the watch-time trap, where optimizing for how long people engage selects for content that holds attention, not content worth attention.

Three structural facts break the naive recipe for a feed.

• The content is produced by the people you rank for. A video catalog is exogenous: it exists whether or not you rank it. Feed inventory is created in response to what the ranker rewarded last week, by users who can see their own metrics. Your ranking function is an input to next week’s candidate distribution.
• The candidate set is a graph query, not a catalog query. There is no global pool of posts. There is your pool: your connections, groups, and follows. It is per-user, small, and it expires.
• The objective is contested. Nobody disagrees about what a good video recommendation is. People disagree, sincerely and permanently, about what a good feed is, and that disagreement shows up as a term in the score.

The feed’s version of the watch-time trap is strictly worse than the video version, for one reason: video supply is a catalog that changes slowly, and feed supply is a population that adapts in days. A watch-time-maximizing video ranker surfaces the worst items in a fixed catalog; an engagement-maximizing feed ranker changes what gets written.

Vocabulary and the model roster

A few words recur throughout.

• Candidate: a post retrieved and eligible to be shown, before anything scores it.
• Retrieval: the step that produces candidates. Cheap, high-volume, allowed to be sloppy.
• Ranker: a model that scores candidates so they can be sorted. The light ranker is cheap and runs on thousands; the heavy ranker is expensive and runs on hundreds.
• Head: one output of a model with several outputs. A model with eleven heads makes eleven predictions from one shared computation.
• Embedding: a fixed-length vector standing in for something non-numeric (a post, author, topic), learned so similar things land near each other.
• Impression: one post shown to one person once. The atomic logged unit.
• Lift: a probability expressed as a multiple of its own base rate. A post with three times the typical comment probability has a comment lift of 3.

The whole system in one paragraph

A viewer opens the app. The system collects a few thousand candidates from four places: posts already delivered into this viewer’s inbox, posts from very large accounts held in memory, posts recommended by search-style indexes, and posts from groups they belong to. A cheap model cuts those few thousand to 600. An expensive model scores each of the 600 on eleven questions (“will they click? comment? hide it? would they say it was worth their time?”). A weight file collapses the eleven numbers into one score per post. A penalty pushes down anything close to a policy violation. A final pass picks 25 posts in order, enforcing variety rules that no single post’s score can express.

Every model in the system

Of the eleven components below, eight are learned models, one is a two-parameter statistical fit, and two are fixed rules. “Offline” runs on a schedule before any request; “online” runs inside the request; “at write” runs once per post when it is created.

Two recurring measures: AUC is the probability the model scores a random positive above a random negative (0.5 is a coin flip, 1.0 is perfect). Calibration means predicted probabilities are true in the long run: among impressions called 3% likely, 3% actually convert.

What the design deliberately does not model

• Long-term user welfare. There is no per-item label for it and never will be, so the weight vector is set by people and audited by slow experiments. Treating it as if a label existed would be the real failure.
• The slate. The score is computed one post at a time, so “three posts from one author in a row” is inexpressible; a separate re-selection pass handles it.
• The supply side’s response. Creators adapting to the ranker is detected and measured, never predicted.
• Per-post engagement rates to decision-grade precision: measuring them would cost several times the platform’s entire daily inventory, so the cold-start path works without them.
• A reinforcement-learning policy over the session, even though the objective is sequential (see Alternatives).
• A large language model on the read path: priced at $3.2 M/day below; such models are confined to offline labelling.

Whose feed is it?

Three parties have a legitimate claim on what the feed maximizes, and they want different things.

Dwell is how long a post stays on screen before the viewer scrolls past. DAU is daily active users.

The first row is the problem: the thing the viewer wants has no logged event. There is no satisfied field on an impression. So the objective gets written in terms of what does have an event, and every failure in this chapter descends from that substitution.

The proxy gap, measured

Bucket a week of impressions by content type and put the engagement rate next to a survey question asked of a sampled subset: “was this post worth your time?”, scored as the share of answers in the top two of a 5-point scale.

Over these eight types (75.5% of inventory), the inventory-weighted any-interaction rate is 0.077 and the weighted satisfaction rate is 44%. The engagement ranking and the satisfaction ranking are nearly reversed at both ends: the Spearman rank correlation between them is −0.55, where 0 would mean unrelated and +1 would mean identical order.

The two highest-engagement categories are the two lowest-satisfaction categories. They are only 8% of inventory today because the current ranker holds them down. Remove the holds and they grow, because they are cheap to produce and win the competition for slots. So if the score contains only engagement terms, its optimum is the bottom two rows of that table (arithmetic, not a slippery-slope worry).

What the objective has to contain

The score has four blocks, each because the ones above it are not enough:

Value(u, p)  =  sum_k  w_k · P_k(engagement type k | u, p)      the actions we can log
              + w_s   · S(worth your time | u, p)               the thing we had to ask for
              - sum_j c_j · N_j(negative action j | u, p)        hide, report, unfollow
              - d(p)  · Value_positive                           integrity demotion

Line 1 is everything the viewer’s behaviour labels for free. Line 2 is the one signal nobody’s behaviour produces, so it must be elicited by survey. Line 3 subtracts “do not show me this”. Line 4 scales the whole thing down when the post looks close to a policy violation. This shape is rewritten correctly, and defended, in the ranking section below.

Candidate generation: a graph query with an expiry date

For a feed the retrieval design inverts twice: the pool is too small, not too large, and it turns over so fast that a standard nightly index build throws away most of the day’s value.

The inventory is small

Count how many posts a viewer is eligible to see. In-network means accounts and groups the viewer has an explicit relationship with; out-of-network is everything else, recommended, not subscribed. A post is useful for a 72-hour (3-day) window, so posting accumulates over three days.

median user: 200 follows × 0.2 posts/day × 3 days  +  45 groups × 1.4/day × 3 days
           = 120 + 189  =  309 posts

Three hundred posts, not a billion. You do not need approximate retrieval over your own network; you can score every eligible post exhaustively. Even a p95 user (900 follows, 200 groups) reaches ~1,380 posts, still a full scan. So the entire ANN apparatus that dominates a video-recommendation design is unnecessary for the in-network half of a feed.

The problem is at the other tail. A new user with 12 follows and 2 groups produces about 16 posts against a 25-impression session, two-thirds of one screen. So the retrieval problem is not “narrow 10⁹ to 10³”; it is fill.

The mixing ratio is per-user and it is the single most consequential dial in the system, because out-of-network content has a different quality distribution, integrity risk, and retrieval architecture. Two sources feed one merged list:

• In-network: an exhaustive scan of a per-user inbox (a list of post ids). No approximation, no recall loss.
• Out-of-network: ANN retrieval over a global embedding index, plus topic and entity inverted indexes (mapping each term to the posts containing it), plus a few hand-tuned sources such as what is trending locally.

The index turns over 33% per day

500 M posts/day × 72 h useful life  =  1.5 B live posts
500 M / 1.5 B  =  33% turnover per day        (a video catalog is ~1%/day)

The standard practice (build the ANN index overnight, serve that snapshot all day) is fatal here. Snapshot at 04:00 and by 20:00 the index has never seen ~22% of the day’s posts. But the loss in engagement is worse than the loss in post count, because the missing posts are the newest, and by the decay curve below that is where almost all the engagement is. Integrating the engagement curve over the first 16 hours versus the full 72-hour window gives about 42% of the day’s engagement made invisible, even though only 22% of the posts are missing.

So the index must accept incremental inserts at sub-minute lag: an HNSW graph (hierarchical navigable small world, a layered graph you walk downhill toward the query) mutated in place instead of rebuilt. Mutation brings tombstone accounting (a deleted item is marked dead, not physically removed, because unpicking it from the graph is expensive) and gradual recall loss as dead entries accumulate (the RAG chapter prices HNSW memory per vector). That cost is what the 33% turnover buys you out of.

Recency, derived rather than assumed

Most feed designs hand-pick a recency penalty. Measure it instead and two surprises fall out: one global number is wrong by a factor of 46 across content types, and once derived it should mostly not be applied as a multiplier at all.

Fit the decay, do not pick it

The naive measurement is confounded: plotting interaction rate against post age across all posts measures two things at once: old posts really do get less engagement, and the ranker shows good posts sooner, so old posts are disproportionately the ones it disliked. Separate them by holding predicted quality fixed: bucket impressions by the ranker’s own score decile and measure age against interaction rate inside one bucket. Any remaining slope is age.

Fitting an exponential rate(t) = rate₀ · exp(−lambda·t) through the endpoints of that curve gives a global lambda ≈ 0.029 /h and a half-life of about 24 hours (the time for engagement to fall by half, equal to ln 2 / lambda).

That single number is wrong for most inventory. Refit per content type:

Breaking news decays about 70 times faster than an evergreen how-to. One global half-life cannot serve both.

The exchange rate, which is what you actually argue about

A decay rate is not a quantity a design review can hold an opinion about. The freshness/quality exchange rate is: if the score is quality · exp(−lambda·t), an old post ties a fresh one when q_old / q_new = exp(lambda·t). At 24 hours:

• Global rate: a day-old post must be 2.0× as good to hold its slot.
• Breaking-news rate: 56× as good.
• Life-event rate: only 1.23× (23% better is enough).

One global half-life applied across that range is off by a factor of 56.2 / 1.23 ≈ 46. In practice it buries a friend’s engagement announcement from yesterday under a fresh aggregator meme and keeps yesterday’s breaking news alive past the point of being wrong. Both are real user complaints and both are the same modelling error.

Do not multiply the decay onto the score

Having measured the decay, mostly do not apply it as a multiplier. Put it in the model’s inputs. Feed age_seconds, log(age), and age_bucket × content_type as features and let the ranker learn the interaction, because:

1. A hand-applied multiplier double-counts: the model already saw age.
2. The right lambda depends on content type, author, viewer, and time of day, exactly the high-order interaction a model fits and a human does not.
3. A multiplier is unconstrained at the tail: exp(−0.168 · 72) ≈ 5.6e-6 zeroes an entire content class.

Keep an explicit decay for exactly one job: breaking ties in retrieval, where there is no model score yet and you must cut a heavy user’s few thousand candidates down to the ~3,000 the light ranker will score. There the multiplier is cheap, monotone, and its errors are recoverable downstream.

The freshness trap

At the young end of the decay curve, where a post is worth the most, the model’s strongest inputs are missing. A five-minute-old post has no engagement counts, so every count-derived feature (post CTR, early like velocity, comment-to-impression ratio) is null.

The obvious fix is “explore: show it and measure.” Price it. To estimate a post’s interaction rate to ±0.02 around a typical p = 0.05 needs about 120 impressions per post (n = p(1−p)/se²), which is roughly 30% of all inventory spent on measurement. And ±0.02 around 0.05 is a 40% relative error that cannot separate a 0.05 post from a 0.06 post, the exact distinction ranking turns on. Ask for four times the precision (±0.005) and the cost, growing quadratically, becomes about 4.75× the platform’s entire daily inventory.

So per-post rate measurement is arithmetically impossible, and the design is forced:

• The cold-start path uses only author-level, content-level, and viewer-author features, no post-level counts. Train it with count features masked so the model does not learn to depend on something that will be absent.
• Exploration is a budget, not a policy: a fixed 3–6% of slots, spent where exploration is decision-relevant. The allocation rule is Thompson sampling: keep a distribution over each author’s true quality and pick each option in proportion to how likely it is to be the best (the reinforcement-learning chapter). Because no affordable budget reaches decision-grade precision, the budget can never be “enough,” so it is spent by an explicit rule instead of being sized to a target.
• Count features enter through a maturity gate: instead of a raw rate k/n from a handful of impressions, show a shrunk estimate (k + alpha·prior)/(n + alpha). When n is small the prior dominates; as n grows the data takes over. The model sees a smooth transition instead of a jump from nothing.

Features

The features group into seven families by where each comes from. In two of them, getting the engineering wrong causes architectural bugs, not accuracy losses.

Content encoders run at write time

Content embeddings are computed at write, never at read. At write, 500 M posts a day is a steady ~5,787 posts/s, an ordinary batch job. At read, the feed serves 200 B impressions a day, so running the same encoders on the read path is 400× the work, all inside a request the viewer is waiting on. Putting a multimodal encoder behind a 400 ms feed request is the most common architectural error in a feed design. Encode once at creation, store the vector on the post, read it back.

Position bias

Position bias is that an item shown higher gets clicked more regardless of quality. A model trained naively on logged clicks learns “was shown first,” not “is good,” and the bias compounds because the model’s output decides tomorrow’s positions. Two standard fixes, derived in the video-recommender chapter:

• A bias tower: a small side-network fed only the position, so the main network is forced to explain everything the bias tower cannot.
• Inverse propensity weighting: weight each logged event by one over its propensity (the probability the old system would have shown that item in that slot) so events the old policy was unlikely to produce count for more.

Two feed-specific twists. The position curve is steeper than a search page’s, because a feed is an infinite scroll and most sessions end before position 20; by position 10 a post gets ~38% of the clicks it would at position 1. But that number entangles two effects (people look less carefully further down, and many sessions simply ended before reaching slot 10) and separating them is feed-specific work. And the propensities exist only if the ranker recorded them, so log the candidate set and score distribution, not just the served slate; without that log, every off-policy estimate is unavailable after the fact.

Multi-task ranking

This is the centre of the design. The ranker is one network predicting eleven reactions, and the harder half of the problem is not building it but turning eleven probabilities into one number, where the product’s value judgement is written down, and where most feed designs go wrong.

The heads

The architecture is a shared bottom (one network body all tasks use, so the expensive representation is computed once) with a small per-task head on top. Eleven heads in four groups:

Those same eleven names are the base-rate table, the scoring config, and the worked bait example below. A head that is not in all of them is a metric somebody wanted, not a head. A head without a measured base rate cannot be given a share weight, and a head without a share weight is one whose contribution nobody can read.

Why one model with eleven heads, not eleven models. The features are identical and the sparse embedding tables (the big lookup tables turning ids like author or topic into vectors) are 95% of the memory. Eleven models means eleven lookups of the same rows; one shared bottom means one. That engineering argument is sufficient on its own. There is also a modelling argument: a plain shared bottom stops being enough once two tasks pull the shared layers in opposite directions. The standard fix is a mixture of experts: several parallel sub-networks with a small per-task gate deciding how much of each expert a task uses, so conflicting tasks quietly stop sharing what they disagree about. The feed-specific note is which tasks conflict: p(comment) and p(hide) peak on the same divisive content, so they are anti-correlated exactly where it matters. Report per-head AUC against single-task baselines; if any head sits more than ~0.005 below its single-task model, the sharing is costing you.

Combining: where the value judgement lives

Eleven calibrated probabilities must become one number. The obvious expression is to multiply each head by a weight and add, subtracting negatives. That is what almost everyone writes, and it is broken in a way no amount of tuning repairs. Four properties explain why.

Property 1: it is linear, so calibration is not optional. In a pure ranking system, any order-preserving distortion of a score is harmless because only the order is used. Here the heads are added, which destroys that safety: if p_comment is 1.4× overconfident, the effective comment weight is 1.4 · w_comment, and nobody wrote that down. In a linear multi-task value model, miscalibration is a silent, undocumented edit to your value judgement. So every head is calibrated independently and reliability diagrams (predicted probability vs observed frequency) are a launch gate. But calibration makes each head’s number true; it does not make two heads’ numbers comparable. That is Property 2.

Property 2: the heads live on base rates 3,000× apart. A head’s base rate is how often that outcome occurs across all served impressions. They are nowhere near each other: 44% of rated posts draw a top-2 survey answer, while 0.015% of impressions draw a report, a factor of about 3,000. For a typical post sitting at every head’s base rate, head k contributes w_k · p̄_k. Multiply the raw weights by their base rates and read the result:

Three things fall out, each contradicting what the weight column appears to say. The survey head is not one term among seven. It is 91.7% of the score. The comment weight looks like 3.7× the like weight (0.55 vs 0.15) but its contribution is 0.70× the like’s, because likes are 5.25× more common; the preference is inverted relative to its author’s intent. And report: 12.0, the largest number in the config, is one of the three smallest terms in the score, because reports run at 1.5 per 10,000 impressions.

None of this is miscalibration. Every base rate is the calibrated truth. The defect is commensurability: a probability of something that happens 44% of the time is being added to one that happens 0.015% of the time, with nothing converting between them. Adding a temperature to a distance is not wrong arithmetic, it is meaningless arithmetic. The reviewer reads w; the ranker obeys w · p̄.

The fix: weight lifts, not probabilities. A lift (the prediction divided by its own base rate) is unitless, which is what makes lifts addable across heads on wildly different scales:

Value  =  sum_k  w_k · (p_k / p̄_k)   -   sum_j  c_j · (n_j / n̄_j)

At a post sitting at every base rate, every ratio equals 1, so term k contributes exactly w_k. The weight vector now is the vector of value shares. Normalize positives to sum to 1.000 and negatives to 0.400, so the base-rate post scores 0.600, the reference every score in this chapter is quoted against. Written back in raw units, the same judgement would require lines like report: −1600 and unfollow: −212. The base rates were always in the weights; they were just never written down, so never reviewed.

Property 3: a share constraint holds at the mean and says nothing about the tail. Ranking is decided by the extremes, not the average post. The engagement bait below draws comments at 22× base rate and scores a quarter of base rate on the survey. The comment term alone contributes 0.150 × 22 = 3.30 to a score whose average is 0.600, while the survey head (nominally the largest share) has 0.350 × 0.25 = 0.0875 to argue with, and loses by a factor of 38. So normalization alone moves the bait from slot 1 to slot 1. The fix is to clip each lift at L_k = p99(p_k)/p̄_k, the point where a head’s outputs stop resembling the data it was fitted on. That lands between 2.8 and 3.4 across the eleven heads, so L = 3 is used for exposition. Above its 99th percentile, a head’s output is far more likely to be solicitation or a calibration tail than genuine quality. Clipping leaves Property 1 intact: below the cap, a 1.4× overconfident head still edits its own weight by 1.4.

Property 4: the weights cannot be learned, because the label is missing. Fitting w would need a per-item label for long-term welfare, and none exists. So w is set by humans and checked by long-running experiments against retention and survey aggregates, a very low-bandwidth channel. A holdout (users kept on an unchanged system) takes 8–12 weeks for a trustworthy read, giving ~5 read-outs per year, and you can run about four in parallel before they contaminate each other: roughly 20 evaluations of w per year, total. A coordinate sweep of n weights costs n evaluations, so 20 evaluations buys two sweeps of a 10-weight vector and not even one sweep of a 40-weight vector. The dimension of the value vector is bounded by experiment throughput, not modelling ability. Keep it at 8–12 terms, in a config file, reviewed by name.

How the weights are actually set, four human judgements and one spend of the scarce experiment budget:

1. Elicit the shares, not the weights. Ask “of the value a post can add, what fraction should come from a person telling us it was worth their time?” The answer is w_survey. This is answerable; the raw-weight version (“0.7 or 0.8?”) is not, because nobody can hold p̄ in their head.
2. Audit the shares by converting back to exchange rates. In raw units the shares assert one comment is worth ~11 likes and one reshare ~19. If the room thinks a comment is worth about three likes, the shares are wrong. People have reliable intuitions about the exchange rate and none about the share.
3. Set the negative block as one fraction of the positive block (here 0.400: an average post loses 40% of what it earns), then split it by the same audit (a report costs ~8× a hide). A post at the lift cap on report alone loses ~72% of the positive block: the negative heads are dormant at the base rate and dominant in the tail, which is exactly what report: 12.0 failed to deliver.
4. Move at most two coordinates per read-out, and label every weight nobody has ever moved as an opinion with a date next to it.
5. Treat re-measuring the base rates as a weight change. p̄ drifts, and re-baselining the denominators silently re-scales every weight, so freeze the base rates for the life of an experiment and review a re-baseline like an edit to w.

Every term has a degenerate optimum

The score is a blend because each single term, maximized alone, has a well-defined worst product.

The last row is why the negatives are a block subtracted from a positive block instead of independent filters, and why guardrails are handled as independent launch blockers, not terms you can trade a good engagement number against. The blend is a set of mutual constraints among terms whose individual optima are each unshippable.

The survey head economics

One head’s labels are not free: sampling 1 session in 20,000 across 8 B sessions, at 30% response and 3 posts each, gives ~360,000 item-level labels a day, against ~2×10¹¹ engagement labels, roughly six orders of magnitude fewer, and they cost real user attention. The reflex is that a head trained on 0.0002% of the data cannot be worth its slot. Both halves of that reflex are wrong.

The volume is sufficient because it is a head, not a model. The survey head does not learn what a post is. The shared bottom, paid for by 2×10¹¹ engagement labels, did that. It only learns a readout: a two-layer tower of ~33,000 parameters. At the usual 10–100 examples per parameter that needs 0.33–3.3 M examples; 360,000 labels a day is ~130 M a year, 40–400× the requirement. The representation is free; only the readout is expensive.

A noisy estimate of the right target beats a precise estimate of the wrong one. Model engagement E and true satisfaction S as standardized variables correlated at rho = 0.35, measured at the item level on the survey-labelled impressions. That is not the −0.55 from the proxy-gap table, which was a rank correlation across content types. Both are real: within a type a more-engaging post is somewhat more satisfying (+0.35); across types the order reverses (−0.55). A relationship that holds inside every group and flips when the groups are pooled is Simpson’s paradox, and it is the whole reason the two signals are not interchangeable.

Suppose true value is V = 0.3E + 0.7S (a stated judgement, not a measurement, and the conclusion is not very sensitive to it). Ranking by engagement alone correlates about 0.639 with V, the number to beat. Ranking by 0.3E + 0.7S_hat, where the survey head’s prediction S_hat has reliability r = corr(S_hat, S), crosses that baseline at:

Break-even is r ≈ 0.49, and a head fit on 130 M labels over a shared representation lands at 0.55–0.65 in practice, comfortably past the line. The 10¹¹ engagement labels do not close the gap because p_click was already estimated far more precisely than any ranking threshold needs. Additional data on a saturated signal has zero marginal value; the first data on a signal you have never measured has enormous value. That asymmetry is the whole argument.

Three practical notes: ask pairwise (“which of these two would you rather have seen?”) over Likert where you can, since Likert scales carry population-specific scale-use bias; correct response bias by reweighting each response by one over the user’s modelled probability of responding, and report weighted and unweighted; and the survey head cannot be the only defense: it is a smooth signal, while the discrete one belongs to the integrity system.

Diversity: the constraint no pointwise score can express

Value is computed one post at a time. It is pointwise. “Three posts from one author in a row” is not a property of any post; it is a property of the slate, the whole ordered page of 25. A pointwise argmax has nowhere to put it, which is why diversity is a re-selection pass over the scored list instead of another term in w.

The pass fills the 25 slots greedily (best post for slot 1, best for slot 2 given slot 1, never revisiting), taking at each slot the post that maximizes Value(p) − sum_c lambda_c · violation_c(p | slate so far), where violation_c counts how far p pushes the partial slate past constraint c. Greedy is not optimal but lands within a few percent of the optimal slate and fits the 6 ms budget.

Eleven percent of pointwise value, measured by logging the counterfactual slate the unconstrained argmax would have produced and reporting the difference. A pass with no measured cost is one nobody can argue with; a pass whose cost is 11% has to justify itself against the 4.9% the ads interleave alone spends.

The author cap is also the only thing that bounds the feedback-loop runaway derived later: a hard cap pins any author’s impression share at 3/25 = 0.12 regardless of affinity, so the self-reinforcing loop saturates. It is a backstop with a number on it, not the real fix. A system sitting against that ceiling has a broken feature definition, which is why the real fix lives in the feature store.

Training

Six decisions, each with a reason. Binary cross-entropy is the standard loss for a yes/no prediction; a logit is the raw score before it is squashed into a probability.

The continual-update loop is where the subtle failure lives: an hourly model trained on the last hour of labels is trained on impressions the previous hourly model chose, 24 rounds a day of a policy training on its own output. The exploration budget is the only source of counterfactual data in a continually-updated ranker; without it, the model’s estimates for anything it currently suppresses are frozen at whatever they were when it started suppressing them.

The integrity interaction

The ranker does not operate alone: the harmful-content system (the content-moderation chapter) produces one number per post, p_violating, and the feed consumes it. The coupling runs in a direction people rarely expect.

An engagement-maximizing ranker is a borderline-content-maximizing ranker

Bucket live inventory by p_violating:

Interaction rate rises monotonically as the probability of violating policy goes up, right to the removal line: 2.4× from the cleanest bucket to the dirtiest servable one, while satisfaction falls from 44% to 14%. This is not a classifier artifact: content that provokes gets responded to, and the policy line was drawn around provocation. So a ranker that maximizes engagement subject to “not removed” has its optimum pressed against the removal threshold from below. Removal alone cannot fix this, because it is a step function and the ranker finds the highest point still on the servable side of the step.

Demotion, and what it buys

The fix is a continuous penalty below the removal threshold:

d(p)   =  1 - 0.9 · clip((p - 0.20) / 0.65, 0, 1)      switches on at 0.20, floors at 0.85
Value' =  d(p) · Value

d(p) runs from 1.0 (no penalty) to 0.10. At p = 0.80, d ≈ 0.169, so the post needs 1/0.169 ≈ 5.9× the Value it had to hold its slot. Now check the penalty against the force it fights: the engagement gradient across the same range is 2.4×, the demotion is 5.9×. Because 5.9 beats 2.4, the optimum moves off the boundary. Comparing the penalty slope against the engagement slope is how you decide whether a demotion curve is strong enough, a question with a numeric answer, not a policy debate.

One subtlety worth stating because it is a real bug: demotion is a multiplier, and multiplying a negative Value by d < 1 shrinks its magnitude, which moves the score up. Applied blindly, the integrity penalty would promote the worst posts. So the demotion is applied only when Value > 0, which keeps the final score non-increasing in p_violating everywhere.

Demotion is how you spend a classifier too weak to remove with

Precision is the share of flagged posts that really are violations; recall is the share of real violations caught. Raising the threshold buys precision and loses recall:

The asymmetry is the cost of a false positive, a legitimate post wrongly flagged. Removing one costs the full post, an appeal, and a trust event; call it R. Demoting it costs roughly 55% of its impressions and nothing else, about 0.09R. That puts the break-even precision for demotion about an order of magnitude below removal, which is exactly why the demotion curve can start at p = 0.20 where precision is only 0.14. Demotion is not a softer removal; it is the mechanism that lets a low-precision classifier do useful work at all. Two mechanisms belong with it: demote the author on a sustained pattern (aggregation gives far better precision than any single post), and never let demotion be invisible: log the counterfactual rank so you can report posts demoted, impressions lost, and the survey delta on affected slots.

Serving: fanout and the celebrity arithmetic

How does a post written once reach millions of candidate sets? Two pure designs each fail on a different tail of the same follower distribution, and the hybrid’s threshold falls out of a memory budget.

Fanout is the one-to-many delivery step; the choice is when to do it. Fanout-on-write (push) does it at post time: append the post id to every follower’s inbox, a per-user list in a key-value store. At 500 M posts/day and a mean of 400 followers, that is 2×10¹¹ writes/day (about 2.3 M/s average, ~5.8 M/s peak), a large but ordinary workload for a sharded key-value store.

The mean is not the problem; the tail is. Split authors into the tiny head of very large accounts and everyone else and the halves come out equal: 5,000 head accounts (4 M followers, 5 posts/day) produce 1×10¹¹ writes/day, the same as the other ~500 M posts combined. The head is 0.005% of the day’s posts and half its writes. And the burst is worse than the average: one post from a 100 M-follower account is 100 M writes, which at ~3 M writes/s is 33 seconds of the entire platform’s write budget. And thousands of such accounts post in correlation because they react to the same events. Fanout-on-write is structurally unable to serve a power-law follower distribution.

Fanout-on-read (pull) does the work at read time: fetch each followed author’s recent posts and merge. At 8 B sessions × 200 authors that is 1.6×10¹² fetches/day, 8× the operations of push, all on the critical path. The latency argument is decisive: if 1% of fetches exceed 20 ms and the 200 fetches are independent, the chance all 200 are fast is 0.99²⁰⁰ = 0.134, so 87% of feed loads hit at least one straggler, and the request’s latency is the maximum over all 200. Pure pull is not viable when 200 things must all come back.

The hybrid

Push the long tail, pull the head. The threshold is chosen so the pulled set fits in RAM on every feed host, which is what removes the straggler term:

~50,000 accounts > 1 M followers × 50 recent posts × 32 B  =  80 MB hot set

Eighty megabytes fits in the memory of every feed server, refreshed by a broadcast stream at sub-second lag. Reading it costs zero network round trips, so the 0.99ⁿ straggler argument evaporates. The merge is a scan over an in-memory array.

flowchart LR
    A["Author posts"] --> D{"followers x posts/day<br/>over threshold?"}
    D -->|"long tail"| W["Fanout-on-write:<br/>append to each follower inbox"]
    D -->|"head account"| H["Broadcast to head-account<br/>hot set: 80 MB, in process"]
    W --> INB[("Per-viewer inbox<br/>sharded KV")]
    INB --> M["Merge at feed request"]
    H --> M
    M --> R["Candidates for ranking"]

Removing the 50,000 hot-set accounts from the push path removes ~55% of writes and 100% of the catastrophic bursts, for 80 MB of RAM per host. Afterward the mean fanout of a pushed post is 180, right at the median follower count (the check that the threshold is in the right place). The trade is this good because of a structural property: the accounts expensive to push are exactly the accounts many people follow, which makes their timelines maximally cacheable. Two details: the threshold is really about followers × posts_per_day, not followers alone (a 200k-follower account posting 40×/day writes more than a 1M account posting twice); and inbox lists are capped (last 500 ids or 72 hours) and materialized only for users active in the last 30 days, which keeps storage around 36 TB (108 TB at 3× replication) instead of ~3× that for inboxes nobody reads.

The request path

flowchart TD
    REQ(["Feed request"]) --> CTX["Session context<br/>viewer features<br/>seen-set bloom filter"]

    CTX --> INBOX[("Push inbox<br/>sharded KV<br/>500 ids · 72 h TTL")]
    CTX --> HOT["Head-account hot set<br/>50k authors · 80 MB<br/>in process · no RPC"]
    CTX --> OON["Out-of-network retrieval<br/>ANN + topic index<br/>incremental insert"]
    CTX --> GRP[("Group / page sources")]

    INBOX --> MERGE["Merge · dedupe<br/>seen filter<br/>~3,000 candidates"]
    HOT --> MERGE
    OON --> MERGE
    GRP --> MERGE

    MERGE --> HARD{"Integrity hard filter<br/>p_violating >= 0.85<br/>blocks · mutes · locale"}
    HARD -->|drop| X(["Removed"])
    HARD -->|pass| LIGHT["Light ranker<br/>3,000 -> 600<br/>2-tower dot product"]

    LIGHT --> HYD["Feature hydration<br/>edge · author · content<br/>post counts w/ maturity gate"]
    HYD --> HEAVY["Multi-task ranker<br/>shared MoE bottom<br/>11 heads · calibrated"]

    HEAVY --> VAL["Value combine<br/>shares x clipped lift<br/>1.000 pos · 0.400 neg"]
    VAL --> DEM["Integrity demotion<br/>d(p) · Value"]
    DEM --> DIV["Diversity pass · greedy<br/>author cap 3/25 · type <=40%<br/>OON floor by segment · ads 1-in-6<br/>~11% of pointwise value"]
    DIV --> OUT(["Ranked feed"])

    OUT --> LOG[("Impression log<br/>position · candidate set<br/>counterfactual rank")]
    LOG --> TRAIN["Hourly continual update<br/>+ weekly full retrain"]
    TRAIN -.-> HEAVY

The diagram above traces one request from top to bottom. Session context is assembled: viewer features plus a seen-set bloom filter, a compact probabilistic structure answering “have we shown this person this post already?” at a fraction of a real list’s memory. Four candidate sources are queried in parallel: the push inbox, the in-process head-account hot set (no network hop, which is why the straggler math does not apply to it), out-of-network ANN and topic indexes, and group/page sources. They merge to ~3,000 candidates with duplicates and already-seen posts dropped. The integrity hard filter removes what must never be shown (p_violating ≥ 0.85, blocked or muted accounts, locale-ineligible posts) before any model spends effort on it. The light ranker cuts 3,000 to 600, feature hydration fills in the real feature values, and the heavy ranker scores them. Value combine, integrity demotion, and the diversity pass turn 600 scored posts into 25 ordered slots. Everything served is logged (position, candidate set, counterfactual rank), feeding the hourly and weekly retraining.

Scale and cost

Everything descends from five traffic numbers: 2 B DAU, 8 B sessions/day (4 per DAU), 200 B impressions/day (25 per session), 8 B feed requests/day (~93 k/s average, ~232 k/s peak at 2.5×), and 500 M posts/day (~5,800/s).

Latency budget (p50 at peak)

auth + viewer feature fetch                        15 ms
inbox read, 500 ids, sharded KV                    12 ms
head-account hot-set merge (in process)             1 ms
out-of-network ANN, 3 indexes, top-200 each        25 ms
dedupe + seen-filter + integrity hard              12 ms
light ranker, 3,000 -> 600                         18 ms
feature hydration for 600 candidates               40 ms
heavy multi-task ranker, 600 candidates            35 ms
value combine + demotion + diversity                6 ms
content payload hydration (text, media, ads)       60 ms
serialization + network                            45 ms
                                                  -------
                                                  269 ms p50   (~430 ms p95 with queueing)

The two biggest items are not machine learning: feature hydration (40 ms) plus content payload hydration (60 ms) is 100 ms, 37% of the budget. Optimizing the ranker from 35 to 25 ms saves 3.7%; batching payload hydration properly saves ~15%. Fix the hydration first.

The constraint is memory traffic, not FLOPs

Sizing the arithmetic is the easy half. The heavy ranker runs a candidate through layers 512 → 1024 → 512 → 256 → 11, about 2.36 MFLOP per candidate, so 600 candidates at 232 k/s peak is ~330 TFLOP/s, roughly 3,000 accelerators at realistic utilization. But that is not the binding constraint. Each candidate needs ~60 sparse embedding lookups (author id, topic, language, …), each reading a row from a big table: 60 × 64 × 2 B = 7.68 KB per candidate, 4.6 MB per request, ~1.07 TB/s of random access at peak. The author embedding table alone is 300 M × 64 × 2 B ≈ 38 GB, too large for one accelerator’s HBM (the fast memory on a GPU), so it shards, and a request’s 600 candidates hit 600 random rows across shards. Feeding 330 TFLOP/s across the network would need ~8.5 Tbit/s of interconnect.

The dense math is embarrassingly parallel; the embedding lookups are a distributed random-access problem. So route each candidate to the shard that already holds its author row, making the lookup local so only the 11 output scores cross the wire, worth more than any architecture change to the ranker. Three standard reductions on top: hash the long tail of author ids into a shared table so rare authors share rows; quantize embeddings to int8 (halves traffic, costs ~0.001 AUC); and cache the viewer-side embedding once per request instead of per candidate.

Metrics

Some measurements gate a launch; most only inform it. Telling them apart matters more on a feed than anywhere else, because the dashboard that says ship is compatible with a system that got worse.

Offline metrics are weak here

A gate blocks a launch; a diagnostic only tells you where to look. On a feed almost everything is a diagnostic. Per-head AUC and NDCG are diagnostics (a head can improve while Value regresses). Per-head calibration is a gate, because miscalibration silently reweights the value model. Correlation of Value with held-out survey score is the closest offline proxy for the thing you care about. Counterfactual off-policy estimates (inverse propensity scoring, doubly robust) are high-variance tripwires, not decisions. Slice metrics on new users, low-connectivity users, and each locale gate on the worst slice.

Offline metrics are weaker on a feed than in almost any other ML system, because the logged data was produced by a policy, the new policy shows different items, and the counterfactual is unobserved for everything not shown. Plan for the online experiment to be the decision from the start.

Online metrics, fastest and least trustworthy first

The metrics you can act on quickly are the ones most easily gamed.

The dashboard that says ship

Ranker v7, a four-week A/B at 2% of users:

sessions per DAU                    +3.1 %      p < 0.001
interactions per session            +5.4 %      p < 0.001      <- everything above ships it
time spent                          +2.2 %      p < 0.001

survey top-2-box                    41.3 % -> 39.1 %   (-2.2 pp, p = 0.004)
hides per 1,000 impressions         2.1 -> 2.5         (+18 %,   p < 0.001)
"see fewer posts like this"         +24 %              p < 0.001
reports per 1,000 impressions       +6 %               p = 0.03
D28 return                          -0.14 %            p = 0.31  (underpowered at 4 weeks)

six weeks after full launch, on the SUPPLY side:
share of inventory classified engagement-bait     2.1 % -> 5.6 %
median posts/day by top-decile authors            3.1  -> 4.4

Every metric that reads out in a week is up; every metric that requires asking a person, waiting a quarter, or measuring producers is down. The last two lines did not exist during the experiment: the supply response is invisible to any A/B smaller than the population, because creators optimize against the ranker most of their audience sees, not the one 2% of it sees. That is not a fixable experimental flaw; it is a property of the system.

What to do given that

• Long-term holdouts. Hold 0.5–1% of users on a frozen ranker for 6–12 months, not to gate launches, but to measure the accumulated delta of everything you shipped, which no individual experiment sees. Expect that aggregate to be smaller than the sum of individual wins; the gap is your budget for novelty (a change looks good just because it is new) and drift.
• Reverse holdouts for supply effects: ramp to 100% and hold out a geography or creator cohort, so the treated population is large enough for supply to respond.
• CUPED to reclaim statistical power (the statistics-and-inference chapter): subtract each user’s own pre-experiment behaviour from their outcome, removing between-user variance. You will need it for the retention arm.
• Guardrails as independent blockers, not terms in a weighted score. Negative-feedback rate, report rate, and survey delta each block a launch on their own, precisely because they must not be traded against a good engagement number.
• Size the retention arm honestly. Detecting 0.2 pp on a 68% D28 base needs about 870,000 users per arm, affordable at 2 B DAU. The binding constraint is duration, not n: the effect takes 8–12 weeks to develop, and you cannot cleanly ship anything else to those users meanwhile.

Failure modes

Each failure gets a mechanism, a detector, and a control, and several are caused by the design working exactly as specified.

Engagement bait amplification

Take a real piece of bait (“99% of people can’t name a country with no letter ‘A’. LIKE if you can. COMMENT your answer. SHARE to challenge a friend.”) and score it. It is at 22× base rate on comments and 0.25× on the survey question. That gap is the whole failure mode.

Score it four ways, switching the two defences (is the survey head in the vector? is any lift clipped at 3×?) on and off. The reference is the base-rate post at Value = 0.600:

Neither fix works alone. Clipping without the survey head takes the bait from 14.3× to 1.8×, better, still top-five. The survey head without clipping takes it to 8.9×, still slot 1, because the survey head has 0.350 × 0.25 = 0.0875 to spend against a comment term of 0.150 × 22 = 3.30, outgunned 38 to 1. Together they reach 0.74×, below the average post: the clip caps what solicited actions can buy, and the survey head, now able to matter because no behavioural term can run away, spends its whole budget against the post. Every probability in that block is already the calibrated truth, and three of the four variants still ship the bait, which is the concrete answer to “why isn’t calibration enough.”

Three defenses, in order of durability: the survey head weighted enough to matter over clipped lifts; a bait classifier as a demotion signal (works, but an arms race: a tax that raises the cost of bait, not a filter that eliminates it); and discounting the solicited engagement types conditioned on solicitation: a like explicitly asked for is worth less than a spontaneous one. The last is the most durable because it attacks the mechanism, not the surface form; the lift cap is its unconditional version.

Narrowing, measured

The filter-bubble claim has a number. Entropy measures how spread out a distribution is, in nats; exp(entropy) gives an effective count: the number of topics the viewer would be seeing in equal proportions.

topic entropy of a user's impressions, 12 weeks on a count-based affinity ranker
week          1     3     6     9    12
entropy    3.41  3.18  2.84  2.51  2.28   nats
eff. topics  30    24    17    12    10    = exp(entropy)

The user did not become less curious. The ranker stopped offering, so the user stopped clicking, so the ranker stopped offering. Topic entropy is the cheapest guardrail in the system and almost nobody puts it on the dashboard.

The feedback loop, formalized

That narrowing is caused by how one feature is defined, not by anything the model learned. For one viewer and author v, take three quantities: affinity a_v, defined the way most systems do it: a decayed count of the viewer’s interactions with that author; impression share s_v ∝ exp(beta · a_v), since the ranker makes share increase with affinity; and the true interest rate r_v, which does not change. Interactions accrue at r_v times impressions and the decayed count settles where accrual balances decay, giving:

a_v  ∝  r_v · exp(beta · a_v)

This is a fixed-point equation with a self-reinforcing right-hand side. Collecting the constants into K, the function g(a) = K·exp(beta·a) − a is convex, so it has a root (a stable fixed point) only when its minimum is at or below zero, which works out to:

a fixed point exists  iff  K ≤ 1 / (beta · e)

The e is the part people drop, and dropping it is not conservative: writing the condition as K·beta > 1 instead of K·beta > 1/e puts the threshold a factor of e ≈ 2.718 too high, so every author whose K·beta lands between 0.368 and 1.0 is called stable and is already running away. Above the real threshold, an author who gets slightly more share accrues more count, which buys more share, and the count has nothing to settle to. Nothing in Value opposes it, because Value is pointwise and this is a slate property; only the author cap in the diversity pass stops it, by pinning s_v ≤ 0.12.

That cap is a real bound and still a patch. Two fixes, the first nearly free:

• Make affinity a rate, not a count: interactions / impressions, shrunk toward a prior. A rate is not mechanically increasing in exposure, and the fixed point dissolves.
• Floor exploration at 4%, so the rate stays estimable for authors the ranker currently suppresses.

Measured with the same effective-count reading:

author entropy of a viewer's impressions, 12 weeks
count-based affinity                3.90 -> 2.40 nats   (49 -> 11 effective authors)
rate-based affinity, 4% exploration 3.90 -> 3.50 nats   (49 -> 33 effective authors)

The feed-specific point is that the loop is closed by a feature definition, not the model (an affinity count is arithmetically increasing in exposure whether or not the model is any good), so the fix lives in the feature store, not in training.

Degradation that writes itself into the model

Under load, the out-of-network retrieval call is the first to time out (it is the slowest of the four sources, 25 ms), and the system falls back to in-network only. That is fine for deep-graph users and catastrophic for the 18% with thin graphs, who need 75–90% of their feed from exactly the source that just disappeared. In one 40-minute incident the low-connectivity segment got 16 items instead of 25, sessions ending within 10 seconds went from 11% to 34%, and next-day return fell 1.8%.

The part that outlives the incident is the training data: every impression logged during those 40 minutes was in-network, so the next hourly update trains on a distribution the system will never see again and mis-ranks out-of-network content for the following hour. Two controls: tag degraded requests at log time and exclude them from training; and when you must degrade, degrade the number of items, not the composition: 15 items with the normal mix beats 25 from one source.

Producer starvation

One failure never shows up in a viewer metric, because it consists of content that was never written. A new author has no history, so author-level features sit at their prior, so they rank low, so they get no impressions, so they never acquire a history. The loop closes on the first pass.

author cohort, first 30 days
                              impressions/post   authors with >= 1 interaction
new authors, no boost                    41                  38 %
new authors, 5% exploration slot        210                  71 %
established authors                     890                  94 %

Thirty-eight percent means six in ten new authors post into silence, and most stop. A 5% exploration allocation on the producer side (a different budget from the viewer-side exploration) raises impressions per post 5× and nearly doubles the share of authors who get any response. This is a supply problem that presents as a ranking metric that looks fine, because the content that would have existed does not exist to be measured.

Summary

Alternatives considered and rejected

Conclusion

A feed ranker is not a video recommender with a different catalog. Its supply is endogenous, its candidate set is a small per-user graph query, and its objective is genuinely contested. The load-bearing ideas:

• Engagement is a measurably bad proxy for value. Across content types the two orderings correlate −0.55, so an engagement-only score has its optimum on the worst content. The survey head is the only direct measurement of the target, and a noisy estimate of the right thing beats a precise estimate of the wrong one.
• Multi-objective weights must be written as value shares over clipped lifts, not raw probabilities. Heads on base rates 3,000× apart cannot be added directly; the base rates were always hidden in the weights, and writing them down is what makes the config reviewable. Clip lifts so the shares still mean something in the tail where ranking happens.
• Recency, integrity, and diversity are each a specific mechanism, not a knob. Fit the decay per content type; replace the removal step with a demotion ramp steep enough to beat the engagement gradient; and enforce slate-level variety in a re-selection pass because a pointwise score cannot see it.
• Serving is a tail problem. Push the long tail, pull the head, and let an 80 MB in-process hot set remove both the celebrity write burst and the straggler latency.
• The most important effects are unmeasurable by an ordinary A/B. The supply response does not occur below population scale, so long-term and reverse holdouts, not a green four-week dashboard, are what tell you whether the system improved.

None of this makes the problem solved. The objective encodes a value judgement no data settles and that you can evaluate only about twenty times a year; the harms are long-term and the experiments short-term; and the system’s training data is its own output. What you can claim is that each failure has a mechanism, each mechanism a detector, and each detector a number attached.

One line to remember: the viewer’s own row has no logged event, so every hard problem in a feed comes from optimizing a proxy for it, and the whole design is the set of defences that keeps the proxy honest.

Further reading

• Ma et al., “Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts” (KDD 2018): the MMoE architecture behind the shared bottom.
• Zhao et al., “Recommending What Video to Watch Next: A Multitask Ranking System” (RecSys 2019): multi-task ranking with engagement and satisfaction objectives in production.
• Covington, Adams, Sargin, “Deep Neural Networks for YouTube Recommendations” (RecSys 2016): the two-stage retrieve-then-rank pattern.
• Malkov, Yashunin, “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs” (IEEE TPAMI, 2018): the HNSW index used for out-of-network retrieval.
• Deng, Xu, Kohavi, Walker, “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)” (WSDM 2013): variance reduction for the retention arm.
• Twitter, “Twitter’s Recommendation Algorithm” (2023 open-source release and engineering post): a public end-to-end feed-ranking stack.

Next: the connection-recommender chapter.