How to design ad click-through prediction

Most ranking systems care only about the order of their scores. This one cares about the scores themselves, because each predicted probability is multiplied by money.

In this lesson, we’ll design ad click-through prediction for a platform serving 2.0e10 (twenty billion) ad requests a day against $32.9B of annual revenue. By the end you’ll be able to:

• Name every model in the ad-serving stack and describe its inputs and outputs.
• Derive the ranking model’s maximum size from an externally imposed latency budget, before quality is even discussed.
• Explain why a probability that is systematically too high breaks the auction, not just the metric, and why the obvious version of that argument is wrong.
• Correct the two biases a real training pipeline injects on purpose: one from downsampling, one from delayed labels.

The one idea

Here, calibration is a correctness requirement, not metric hygiene.

In a search ranker, a miscalibrated score changes nothing: the order is preserved, so nothing is lost. In an auction, the predicted probability is multiplied by a bid to form a price, compared against a fixed reserve, and fed to a budget controller.

A number that only has to be in the right order is a rank. A number that gets multiplied is a price. Everything distinctive about this problem follows from that.

flowchart TD
    R["Raw ranker score<br/>(downsampled world)"] --> C["Calibration layer<br/>ln(w) offset + per-segment isotonic"]
    C --> P["Calibrated P(click)"]
    P --> RANK["Ranking: pick largest eCPM<br/>order only — safe under rescaling"]
    P --> PRICE["Pricing: divide by p<br/>for a per-click price"]
    P --> FLOOR["Floor / CPM check<br/>eCPM vs a fixed number"]

Only the first consumer, ranking, is invariant to a monotone rescaling of p. A bias a ranker would never notice lands directly on pricing and on the floor comparison, both of which cost money. A separate constraint (the auction allows roughly 20 ms to score a hundred candidates) sets the model-size ceiling before architecture is discussed.

Notation you need first

Scientific notation. 2.0e10 means 2.0 × 10¹⁰. 5e-2 means 0.05. Volumes here span nine orders of magnitude, so writing zeros out is unreadable.

Odds and log-odds. If the probability of a click is p, the odds are p / (1 - p), running from 0 to infinity. The log-odds or logit is ln(p / (1 - p)). This is the scale a logistic model works in: the raw number a logistic regression or neural network emits is a logit, converted to a probability by the sigmoid p = 1 / (1 + e^-z).

Why it matters: several distortions in this lesson multiply the odds by a constant. Because ln(a·b) = ln(a) + ln(b), multiplying odds by a constant is adding a constant in log-odds. That is why every correction below turns out to be “add one number to every score” instead of anything shape-changing.

Calibration means that among all the times a model says 0.02, about two in a hundred really click. It is separate from ranking: a model can order every candidate perfectly and still be double on every score. See the calibration and drift chapter for the general treatment; here is where it becomes money.

Prior shift is what happens when the base rate in your training data is not the base rate in production, because you threw away most negatives, or because a label had not arrived yet. In log-odds the correction is a single constant added to every score.

The models in the stack

An ad-serving stack is not one click model. There are two rankers in a cascade, a calibration layer between the ranker and the money, and four smaller models that correct biases the logs cannot avoid. The overall shape: one model produces the score, one fixes the score, and four correct biases introduced by logging.

Two points stand out. The heavy ranker’s output is not the system’s output; the calibration layer stands between them, and every money argument here is about that layer, not the network. The last four rows are bias corrections, not accuracy improvements, because the click log contains only ads that were shown, in slots users may not have looked at, with conversions that may not have arrived yet.

Framing

The vocabulary of ad serving

eCPM matters most. A CPM advertiser already bids in eCPM: they named a price per thousand impressions. A CPC advertiser bids per click, so to compare them you must predict how many clicks a thousand impressions produce, and that prediction is the model’s output:

eCPM = 1000 · p · bid = 1000 · 0.018 · $1.20 = $21.60 per 1,000 impressions

The model’s error goes directly into that dollar figure.

The problem, in numbers

Every “this costs X% of revenue” claim below is $32.9B/yr times a percentage.

Clarifying questions that change the design

Two answers are load-bearing. CPC and CPM advertisers compete in the same auction: if every bidder went through the model, a uniform bias would cancel and calibration would barely matter, but because some bids are known exactly and some predicted, there is nothing for the error to cancel against. The 20 ms scoring budget is also imposed from outside, a slice of an exchange’s 100 ms wall clock, not negotiable by the team, and it sets a hard parameter ceiling before quality is discussed. The 1% base rate, $0.45 CPC, and 500M DAU only scale the dollar figures; no design decision flips if they move 30%.

The ML objective

The loss: log loss, and why not a ranking loss

The setup is pointwise binary classification: score each (request, candidate) pair on its own, label 1 for click and 0 otherwise, train with log loss.

L = -(1/N) · sum_i [ y_i·log p_i + (1 - y_i)·log(1 - p_i) ]

When y_i = 1 the penalty is -log p_i (zero if you said 1.0, unbounded as you approach 0); when y_i = 0 it is -log(1 - p_i). Averaged, it is the penalty for confident wrongness.

Log loss is a strictly proper scoring rule: its unique minimizer is the true conditional probability, and reporting anything else scores worse in expectation (proper scoring rules). That is exactly the guarantee an auction needs, because the auction consumes the number, not the order.

A pairwise ranking loss has no such property: its minimizer is any monotone transform of the truth: double it, square it, take its log, and the ordering is unchanged. That is precisely the family of transforms that costs money here, which disqualifies LambdaRank-style objectives for this problem.

What the auction does with p

A reserve price (floor) is the minimum eCPM a publisher accepts. A generalized second-price (GSP) auction charges the winner what it would have taken to beat the runner-up, which is why the runner-up’s eCPM (eCPM_2) appears in the price:

eCPM_i  = 1000 · p_i · bid_i      CPC advertiser
eCPM_j  = bid_j                   CPM advertiser (no model)
show if max_i eCPM_i >= floor
price_1 = eCPM_2 / (1000 · p_1)   per-click price (unit conversion: eCPM is per 1,000 impressions)

For example, a runner-up eCPM of $21.60 and a winner rate of 2.092% gives 21.60 / (1000 · 0.02092) = $1.03 per click.

The number feeds three consumers: ranking picks the largest eCPM, pricing divides by p, and the floor comparison tests eCPM against an absolute number the model has no part in. Only ranking is invariant to a monotone rescaling of p. A bias a ranker would never notice lands directly on the other two.

Why calibration is non-negotiable here

One claim carries the lesson: a miscalibrated click model breaks the auction itself, not just the metric that measures the model.

The naive argument, and why it fails

The standard answer (“the wrong ad wins”) does not survive scrutiny. Suppose every predicted CTR is inflated by the same factor k = 1.25. Multiplying every score by the same number cannot change their order, so ranking is untouched. And the GSP price is a ratio: inflating every prediction by k multiplies both the runner-up’s eCPM (top) and the winner’s rate (bottom) by k, and they cancel.

honest:        price = 21.60 / (1000 · 0.020)          = $1.080
inflated 1.25x: price = 27.00 / (1000 · 0.025)          = $1.080   ← identical

A uniform multiplicative bias cancels exactly in a second-price auction between two ads scored by the same model. That forces the question to the places where the bias does not cancel.

Three places the bias does not cancel

The cancellation worked only because both sides of every comparison came out of the same model. The three failures are exactly the comparisons where that is false.

(a) The reserve price is an absolute number the model is not in. The floor is a figure the publisher typed into a config; it does not move when your model moves. An ad worth $7.20 (below an $8.00 floor) but predicted 1.25× high shows at $9.00, and the publisher realizes $7.20 on a slot they priced at $8.00. Roughly 6% of auctions sit within 15% of the floor, so this leak is concentrated exactly where publishers are most price-sensitive.

(b) CPM advertisers do not go through the model. A CPM advertiser states an eCPM directly; no model touches it. Every mixed auction therefore compares a predicted number against a known one, and the damage runs both ways:

• Under-prediction (0.85×) on a $21.60 CPC ad hands the slot to a $19.00 CPM bid, a 12% loss on those impressions.
• Over-prediction (1.25×) lets a $15.60 ad displace a $17.00 CPM bid worth more, an 8.2% loss.

There is no direction of error that is safe.

(c) The auto-bidder consumes the probability directly. Target-CPA auto-bidding sets bid = target_CPA · pCVR, and pacing spreads a daily budget by forecasting sum of p · price. Neither compares p to another model’s output; both use it as a number in its own right. Inflate p by 1.25× and the pacer believes the budget will exhaust 25% early, so it throttles and the advertiser under-delivers. That surfaces as a sales escalation, not as anything on the ML dashboard.

The real cost: heterogeneous error and the optimizer’s curse

Real calibration error is segment-structured, not uniform. A segment is a traffic cell, say “mobile · gaming · slot 1 · 9pm.” Any one cell can be systematically over- or under-predicted while others are fine. The uniform bias that cancelled above is a fiction.

And an auction does not average over candidates; it takes the maximum over 100 at a time. When you pick the highest score, you disproportionately pick a candidate whose error happened to be positive: high-error-in-your-favour and high-true-worth both push a candidate to the top and you cannot tell them apart. This is the optimizer’s curse: whatever you pick as best is, on average, worse than your estimate said, and the gap grows with the number of candidates.

To price it, model each candidate’s log true worth as v_i ~ N(0, s_v²) and the model’s score as m_i = v_i + e_i, with e_i ~ N(0, s_e²). Here s_e is the size of the model’s calibration error in log-odds, the number this whole subsection is about. The auction shows argmax_i m_i; realized revenue is E[exp(v_selected)], and the ideal is E[exp(v_best)]. Their ratio has no clean closed form (both the max-of-n formula and exponentiating the log-gap overstate it), so it is estimated by Monte-Carlo simulation with n = 100 candidates and s_v = 0.6 (a 1.8× eCPM spread per standard deviation).

The last column is one multiplication: at s_e = 0.25 the auction leaves 9.0% on the table, and 0.090 × $32.9B = $2.97B. Halving the log-odds error from 0.25 to 0.10 recovers 7.4 points of revenue, about $2.42B a year on this traffic. AUC averages over all pairs; the auction consumes only the maximum of 100. Those are different statistics, and only one of them pays: a monotone rescaling that leaves AUC bit-for-bit identical moves this table by a full column.

Two properties of s_e shape the fix. It has two components: refinement error (genuine ignorance about which impressions convert, curable only by better features or more data) and reliability error (the score being systematically mis-scaled, removable cheaply by fitting a 1-D map on held-out data). The calibration layer attacks reliability. And systematic error is more expensive than random error: random error averages out over many auctions, but a systematically over-predicted segment wins auctions persistently, so its losses accumulate in one direction. That is why the fix has to be per-segment.

The production instrument: COPC, per segment

COPC is clicks over predicted clicks:

COPC = observed clicks / sum of predicted probabilities

over some slice of traffic. COPC = 1.00 is calibrated; > 1.00 means the model under-predicts; < 1.00 means it over-predicts.

Global COPC is nearly useless, because the errors cancel. Five segments miscalibrated in opposite directions can add up to a global 1.003 that looks perfect:

The weighted contributions (share × (COPC − 1)) sum to +0.003, so global COPC is 1.003, while the segments that actually compete against each other run from 1.34 down to 0.66, a 2× spread. The weights are shares of predicted clicks because COPC is a ratio of totals, so each segment enters in proportion to its contribution to the denominator.

Two rules follow. Alert on per-segment COPC against a fixed grid (vertical × device × slot × hour bucket), never on the global number. And treat the dispersion of segment COPC (the standard deviation across cells) as the headline metric, because it is a direct estimate of the s_e the dollar table is priced against. That measurement (that the error is segment-structured, not random) is the section’s real claim, and it is why the fix below is per-segment.

Data and labels

The label here is unusually good (free, instant, and produced twenty billion times a day), which makes it easy to miss that it is also biased in three specific ways. An attribution window is the period after showing an ad during which a later action is credited to it.

One training row:

So 99 rows in 100 are negatives, which is what the downsampling section is about.

Three properties of this label

1. Cheap, immediate, enormous. Nobody labels by hand; the user does, twenty billion times a day. There is no delay for clicks. For conversions there is, and it is worse (see delayed conversions below).
2. Conditioned on having been shown. Only shown impressions produce rows, so the training distribution is a snapshot of what the current ranker already believes, not a sample of the world (the feedback loop below).
3. Contaminated by placement. A user must look at an ad before clicking, so a raw click measures examination times relevance, and the log records only the product (position bias below).

Joining impressions to clicks

An impression at time t and a click at t + 12 s arrive from separate services, so the join must wait a bounded interval before declaring an impression un-clicked. That bound is a watermark: a timestamp the stream processor treats as “everything older has certainly arrived.” Too short drops late clicks; too long makes training data stale.

Two failure modes push the same way: duplicate impressions inflate the denominator, dropped clicks shrink the numerator, and both bias measured CTR down. Reconcile the stream join against a daily batch recomputation and alert above 0.2% divergence. A silent 1% click loss is indistinguishable from a 1% CTR decline and gets debugged as a product problem for a week.

Train/serve skew and point-in-time correctness

One bug produces more silent calibration damage than any other, because it is a property of how a row is built. Train/serve skew is any difference between how a feature is computed at training time versus serving time. Point-in-time correctness is the discipline that prevents it: every feature on a training row must be computed from only the information available at the instant the impression was served, never from the complete history an offline job happens to have.

This bites hard because the design puts counter features on the request path: running totals like an ad’s own impressions and clicks, or a user’s click counts over 1/7/30 days. A running total is exactly what an offline job computes wrong by default: at serve time a live snapshot lags the click stream (say 90 s behind, reading 1,180 clicks), while the next-day offline job grouping the click table “before that timestamp” sees more (1,206 clicks, the lag plus late-arriving rows). The model is fitted on a counter no serving process can produce.

That gap is dangerous because it is systematically upward at train time (offline always has more history), largest on the fastest-moving ads, and varies by vertical, so it lands squarely in the expensive, segment-structured s_e, not in a constant an offset could absorb. Three controls, none optional:

1. Log the feature vector that was actually served and train on it, instead of recomputing.
2. Where a feature must be recomputed, compute it as of the impression timestamp minus the pipeline’s measured lag.
3. Alert on the distribution of every feature, serve-side against train-side, the only detector that fires before revenue does.

Extreme imbalance, negative downsampling, and the correction it forces

CTR is ~1%, so 99% of rows are negatives. You should not fix this for statistical reasons: log loss is proper so it already wants the true probability, the consumer is a probability not a class label, and there is no threshold to move. You downsample negatives for exactly one reason: compute.

Keep every positive; keep each negative with probability w = 0.05. Starting from 1,000,000 rows, 10,000 positives are all kept and 990,000 negatives become 49,500, giving 59,500 rows, a 16.8× smaller training set. But the base rate inside that data is now 10,000 / 59,500 = 0.16807. The model will learn to predict around 16.8%, not 1%, and nothing in the loss knows that is wrong.

The correction

Downsampling throws away negatives only, and uniformly at random. So the feature distribution among positives is untouched, and among negatives it keeps its shape but its total mass is multiplied by w. The ratio of the two class-conditional densities is therefore unchanged, and only the mixing proportion moves. Writing the odds in both worlds, everything cancels except the prior-odds ratio, which equals w exactly:

o = w · o'         (true odds = w × sampled odds)
logit(p) = logit(p') + ln(w) = logit(p') - 2.9957

Downsampling negatives is an intercept bug and nothing else, one number added to every score. AUC is bit-for-bit unchanged, the ranking is untouched, and yet every probability is wrong by a factor of 1/w = 20 in the odds. The odds error is exactly 20× on every row, but the probability error is not uniform:

A downsampled model that says 0.95 actually means 0.49. Feed that into eCPM = 1000·p·bid and you have inflated the ad’s value ~1.95×, which by the three-places argument is money handed from the publisher to that advertiser on every mixed auction. The price error shrinks toward the top of the range, where the expensive impressions are, but most mass sits near the 16.8% prior, so the typical impression is mispriced ~16.8×.

The whole correction is two lines of code:

from math import log

def downsample_correct(p_sampled, w):
    """Undo negative downsampling at rate w (positives kept at rate 1).
    Exact only when negatives are sampled uniformly at random."""
    odds = p_sampled / (1.0 - p_sampled) * w
    return odds / (1.0 + odds)

def logit_offset(w):
    """The same correction as a constant added to every logit."""
    return log(w)          # ln(0.05) = -2.9957

What downsampling costs, statistically

Throwing away 94% of the data costs only ~9% in standard errors. Fisher information measures how much a dataset tells you about a parameter; for a logistic model the per-row factor that matters is p(1 - p), which is largest at p = 0.5 and collapses toward the ends. A near-certain non-click (p ≈ 0.01) teaches you almost nothing.

Moving the base rate from 1% to 17% makes each row ~14× more informative (p(1-p) goes from 0.0099 to 0.140), which nearly pays for the 16.8× fewer rows:

information retained = 14 × 0.0595 ≈ 0.84    → 84% of the information on 5.95% of the rows
standard errors inflate by 1/sqrt(0.84) ≈ 1.09   → 9% wider

The three ways teams get this wrong

1. Forgetting the offset. Invisible: AUC is unchanged, and log loss on the downsampled validation set looks fine because it has the same wrong base rate. Nothing fails until revenue does. Fix: validate log loss and COPC on an un-downsampled holdout.
2. Double-correcting. Applying ln(w) and then fitting an isotonic map on downsampled data corrects the same bias twice. Fix: apply the offset first, calibrate on true-prior data second.
3. A segment-varying w with a global offset. Tempting: keep all negatives on rare inventory, downsample hard on the head. But the moment w varies by segment and the offset does not, each segment is shifted by the wrong constant, manufacturing exactly the segment-structured error the optimizer’s-curse table prices at billions. Fix: keep w global, or carry it per row and apply ln(w_i) per row. This one survives code review.

The calibration map, fit per segment

The ln(w) offset fixes the one bias that is provably constant in logit space. It does nothing for the segment-structured error (the vertical that runs hot, the daypart that runs cold) because that error is not a single number. The fix is a per-segment monotone recalibration:

• Fix a segment grid in advance (vertical × device × slot × hour bucket).
• Fit one 1-D map from predicted probability to observed rate per cell.
• Fit it on that cell’s own un-downsampled holdout, with the ln(w) offset already applied.

The map is isotonic regression, an order-preserving (never downhill) step function, fit by pool-adjacent-violators (PAVA): sort predictions ascending, find two neighbouring groups whose observed rates go the wrong way round, merge them into one block at their combined average, and repeat until nothing is out of order.

def isotonic_fit(pred, label):
    """PAVA: the non-decreasing map from predicted probability to observed rate.
    Returns (edges, values): a step function where values[j] applies to every
    prediction <= edges[j]. Fit per segment on an un-downsampled, offset-corrected holdout."""
    order = sorted(range(len(pred)), key=lambda i: pred[i])
    out = []
    for i in order:
        out.append([float(label[i]), 1, pred[i]])        # [sum_y, count, x_right]
        while len(out) > 1 and out[-2][0] / out[-2][1] >= out[-1][0] / out[-1][1]:
            sy, c, xr = out.pop()                         # violation: merge blocks
            out[-1][0] += sy; out[-1][1] += c; out[-1][2] = xr
    return [b[2] for b in out], [b[0] / b[1] for b in out]

Non-decreasing is the point: it repairs calibration without ever reversing two candidates inside a segment. But be precise about the cost. PAVA works by merging blocks, so it manufactures ties the raw score did not have: on one measured cell, 60,000 distinct raw predictions collapse to 56 distinct calibrated ones, 4.5% of strictly ordered pairs come back tied, and 18% of 100-candidate auctions end with a tied top. Grouped AUC moves about −0.0006, small but not “untouched.” The honest claim is that isotonic never reverses a pair, at the cost of tying some. The operational consequence is one line of serving code: break ties with the pre-calibration score, the finest ordering you have, instead of by whatever order the candidate list arrived in.

Two cheaper alternatives fail. Temperature scaling divides every logit by one learned constant. One parameter cannot carry per-segment corrections. Platt scaling fits a logistic curve, imposing an S-shape the segment biases have no reason to follow.

In practice, fitting one isotonic map per cell collapses a ~2× pre-calibration COPC spread to inside [0.98, 1.03], with dispersion ~17× smaller. Since dispersion is the direct s_e estimate, that is the move from the s_e = 0.25 row of the dollar table to below the s_e = 0.10 row, the row pair worth $2.42B a year. Three details decide whether it works in production: the grid must be coarse enough that every cell fills (fall back to the parent cell below a traffic floor), the map must be refit on the model’s cadence, and the auction must carry and break ties on the pre-calibration score.

The load-bearing assumption of the whole downsampling correction is the one the code’s docstring states: negatives must be dropped uniformly at random. Sample them non-uniformly (keep only top-slot impressions, say) and the distortion is no longer a constant in log-odds, and no single offset can undo it.

Features: high-cardinality categoricals at billions of values

The dominant problem is cardinality: the number of distinct values a field can take. An embedding is a short learned vector standing in for a categorical value so a model that only does arithmetic can work with “advertiser 4,182,113.”

Each value gets 16 numbers at 4 bytes each, so 64 bytes per row. The naive table would cost ~148 GB, and one field is 86% of the bill: the user-id table alone is 128 GB. That is where the fix starts.

Do not embed the user id

The mean user has 900 impressions and 9 clicks over 90 days, enough to learn a vector from. But that mean is a lie, because impression counts are Zipfian (a few users carry most volume, a long tail carries almost none). The median user has ~40 impressions and zero clicks.

That is fatal. Gradient descent updates an embedding only when a row containing that value appears, and pushes it upward only on a label of 1. With no positive labels, the vector receives no positive gradient; the only thing pulling on it is the regularizer, which decays it to the prior. For the median user, a 16-dim identifier embedding converges to the prior and carries no information.

Replace it with user history features, aggregates instead of identity:

• Click counts and recency by category, advertiser and vertical over 1/7/30 days.
• A pooled embedding of the last 50 ads the user interacted with.
• The user’s own CTR shrunk toward a device-and-geography prior.

That removes 128 GB (86% of the footprint) and improves the median user’s prediction, because a feature that generalizes beats an id that does not. A user with 40 impressions and no clicks still has a device, geography and category history, all of which transfer from other users; their id transfers from nobody.

Hashing, and why the collisions land where they don’t matter

Hashing a feature runs its value through a hash function and uses the result as an index into a fixed-size table, no dictionary to build or keep in sync. The price is collisions: two ids sharing one row and one embedding. Hash the remaining ids into 2^24 = 16.8M buckets.

The naive objection (“5e7 ads into 1.7e7 buckets is ~3 ads per bucket”) is true and irrelevant, because impressions are Zipfian: almost all three-way collisions involve ids nobody ever sees. What matters is collisions among the head, the ~5.0e5 ads carrying ~60% of impressions. The standard birthday-collision approximation gives:

P(a head ad shares a bucket with another) = 1 - exp(-5.0e5 / 1.68e7) = 0.0294

So 2.9% of head ads collide with another head ad; the rest is tail-on-tail, and a tail id’s embedding was going to be noise anyway. The damage is even asymmetric in a helpful way: a bucket’s weight is dominated by whichever feature occurs more, so a head-tail collision leaves the head correct and gives the tail the head’s prior, a better estimate than it had alone.

For the 2.9%, use a frequency-based hybrid table: dedicated rows above a 10,000-impression threshold (~1.2e6 ids), hashed shared rows below. That drives the head collision rate to 0 and costs 1.2e6·64B + 2^24·64B ≈ 1.15 GB, against the 148 GB the naive table cost. Two operational details: re-evaluate the threshold as ids churn (40% a month), and reset a bucket’s embedding whenever its id set changes, so a retired ad’s weight never becomes a new ad’s starting prior.

The load-bearing assumption of this whole section is that impressions are Zipfian. It is why a hashed table works (collisions concentrate in a noisy tail) and why the user-id embedding is worthless (the median user has no clicks). If traffic were uniform, both arguments reverse.

One row, scored end to end

Here is one populated candidate carried through the pipeline. Its heavy-ranker output is a logit in the downsampled world, the 16.8%-base-rate world, not the real one:

1. Sum the contributions into a raw logit: intercept −1.85, wide cross (advertiser × domain) +0.62, wide cross (vertical × hour) +0.18, deep term +0.20 → −0.85.
2. Sigmoid → p_sampled = 0.29943, a probability in a world where 95% of negatives were thrown away, so not a probability of anything real.
3. Apply the ln(w) offset → p_true = 0.02092 (the 0.300 → 0.021 row of the correction table).
4. Run the auction: eCPM = 1000 · 0.02092 · $1.20 = $25.11. It beats ad A’s $21.60, clears the $8.00 floor, and wins at a GSP price of $1.03 per click against a $1.20 bid.

Now forget the offset. The same candidate posts an eCPM of $359.32, 14.3× too high (the price error, not the flat 20× odds error). It beats every CPM bid it meets, including ones worth more; the publisher promised $359.32 realizes $25.11; and the advertiser is not overbilled, because GSP divides by the same inflated p, so their price per click falls. No party complains, so the leak persists for months. That single row is the whole lesson: the same model, ranking and AUC, and a fourteen-fold error in the only number anybody prices against.

Models: LR, FM, deep, and what each one fixes

The ranking model is built in steps, each fixing a named limitation of the one before.

flowchart TD
    LR["Logistic regression + hand crosses<br/>memorizes only observed pairs<br/>mute on ~99% of possible pairs"] -->|generalize to unseen pairs| FM["Factorization machine<br/>pair weight = &lt;v_i, v_j&gt;<br/>each feature learns from all its rows"]
    FM -->|field-specific interaction| FFM["Field-aware FM<br/>one vector per field met"]
    FM -->|higher-order interaction, but blurs| DEEP["Deep MLP<br/>any-order interactions<br/>poorly calibrated"]
    FFM --> WD["Wide & Deep<br/>linear crosses memorize +<br/>MLP generalizes"]
    DEEP --> WD

Logistic regression on hand-built crosses, and its exact limit

A cross is a feature made by pairing two others: “advertiser 4182 and domain 91177” as a single value with its own weight. Logistic regression is linear in the features it is given, so it has no other way to say a combination behaves differently from its parts; you must build the cross by hand.

Of the 5e6 × 1e7 = 5e13 possible advertiser-domain pairs, even 1.8e12 training rows observe only ~4.0e11 (0.8% coverage) and most pairs are unobservable in principle, since there are 28× more pairs than rows. A cross weight is learnable only for pairs you have observed; for every other pair it contributes exactly zero. LR with crosses is a memorization device: excellent on the head, mute on everything unseen, with parameters growing as the product of cardinalities.

Factorization machines: what they actually fix

A factorization machine keeps LR’s structure but changes how pairs are represented. Instead of one free weight w_ij per pair, it gives every feature a short vector v_i and computes the pair’s weight as the inner product <v_i, v_j>:

y = w_0 + sum_i w_i x_i + sum_{i<j} <v_i, v_j> x_i x_j

The advertised win is parameter count: n·k instead of n²/2, for n = 2.05e9, k = 16, that is 3.3e10 versus 2.1e18, a factor of 64 million. But the real win is statistical. v_i receives gradient from every row containing feature i, not only the rare rows containing the specific pair (i, j). So a pair with zero co-occurrences still gets a non-trivial prediction, estimated from n_i + n_j observations instead of 0. That is exactly the generalization to unseen crosses LR structurally cannot do. It fixes the 99.2% of pairs LR is mute on. FMs are also cheap to score: the pairwise sum has a closed form that sweeps the features twice per dimension instead of enumerating pairs (1,440 multiplications instead of 15,840 for 45 active features, ~11×).

FMs still cannot do two things. They are degree-2 (pairs only; three-way interactions are invisible), and each feature carries one vector regardless of what it is meeting, yet “user × hour” and “user × publisher” plausibly need different aspects of the same user. Field-aware factorization machines (FFM) fix the second by giving each feature a separate vector per field (a whole column, e.g. “publisher”), at F× the memory and ~4× the scoring time.

Deep, and why the wide part does not go away

A multi-layer perceptron (MLP) over concatenated embeddings learns interactions of any order, but through a smooth function approximator, and smoothness is exactly wrong for memorizing that one advertiser-domain pair that converts at 8× the base rate, because smoothness spreads that fact over its neighbours. Deep models generalize and blur; linear crosses memorize and do not generalize. Wide-and-deep exists because those are different jobs, not because two models beat one.

Two metrics decide the choice. RIG (relative information gain) is 1 - logloss / H, the fraction of the label’s uncertainty the model removed, where H = 0.056 nats is the entropy of a 1% coin. ECE (expected calibration error) buckets predictions, compares each bucket’s mean prediction to its observed rate, and averages the gaps, the direct measurement of the property worth billions. Lower is better for log loss and ECE; higher for RIG.

DCN v2 (Deep and Cross Network v2) builds crosses up to a chosen degree inside the network, so you get the memorizing half without hand-writing 40 crosses.

The ECE column is the one that decides this table. “Deep only” beats “FM” on log loss (by 0.00017) and RIG (by 0.30 points) but is 2.7× worse calibrated (0.0071 vs 0.0026). Deep nets trained to convergence on cross-entropy are systematically overconfident (why deep nets are overconfident). By the dollar table, that ECE regression costs more revenue than a 0.3-point RIG gain earns. A model that wins log loss and loses calibration is a losing model here, and log loss alone will not tell you.

One last intuition: RIG going from 1.02% to 8.38% and log loss going from 0.05543 to 0.05131 are the same fact stated twice. At a 1% base rate the entropy is 0.056 nats, so every log loss is pinned near 0.056 and the fourth decimal is the whole game. That is why you report RIG, not raw log loss. But the honest launch gate is not ECE either. It is per-segment COPC dispersion, since a uniform ECE shift would cancel in the auction.

Training: online learning, and the freshness/stability tradeoff

The distribution moves hourly: campaigns launch and exhaust budgets, creatives rotate, news shifts the traffic mix, the time of day changes who is on the platform. A model retrained once a day is stale by the time it deploys. The answer here is continuous training, which has to be earned.

FTRL-Proximal, and the two things it buys

FTRL-Proximal (Follow The Regularized Leader, proximal variant) is an online optimizer that updates one example at a time. It earns its place for two properties, neither of which is “it converges faster.”

Per-coordinate learning rates. Each feature’s rate is alpha / (beta + sqrt(sum of its past squared gradients)), so the more a feature has been updated, the smaller its rate. Each feature anneals on its own schedule. That matters because feature frequencies span nine orders of magnitude: a single global rate would either freeze the tail or make the head oscillate. A head feature (1e10 occurrences) ends up taking ~18,000× smaller steps than a 30-occurrence tail feature, automatically.

Exact zeros. L1 regularization penalizes the sum of absolute weights and, unlike L2, drives them all the way to zero. FTRL’s update produces true sparsity: a weight that never earns its place is never materialized: no serving row is allocated at all. That takes 4.1e9 touched buckets down to 3.2e8 nonzero weights, or 1.3 GB instead of 16.4 GB. This is a deployment constraint expressed as a regularizer, not a generalization argument (which would be marginal at 1.8e12 rows).

What freshness is worth

Run a permanent staleness holdback: a small traffic slice served by a deliberately frozen model, refreshed weekly. It must be online, because offline log loss on a frozen test set understates staleness: the test set cannot show you the traffic the stale model has never seen.

RPM is revenue per mille (per thousand impressions). A 24-hour-old model costs 5.1% of revenue, 0.051 × $32.9B = $1.7B a year. That number, not a taste for streaming, is what justifies the operational cost of online training.

The stability side, and the incident it prevents

Online learning has no epoch (no pass over a fixed dataset that ends at a known point) and therefore no natural rollback point. A corrupted hour is simply absorbed into the weights. The classic incident is a train/serve skew: an upstream logging change drops the slot_position field in one region, feature assembly defaults it to 0 (“slot 1”), and online updates begin attributing slot-1 examination to ad relevance. COPC in that region climbs to 1.31 before a guardrail fires ~40 minutes later.

The damage estimate: EU is 22% of a $90M/day platform, so an 86-minute incident exposes ~$1.18M; a COPC of 1.31 is ln(1.31) = 0.27 of segment-structured log-odds error, which by the dollar table costs 10% of RPM on that traffic, or **$121k**. Four controls, in order of value:

1. Checkpoint every 15 minutes, keep 48 (12 hours). Without a snapshot there is nothing to roll back to.
2. Keep a batch-trained model live as a fallback, so there is somewhere to send traffic while you debug.
3. Guard the input batch. Reject any update whose positive rate, missing-feature rate, or number of distinct feature values deviates more than 5 standard deviations from the trailing hour.
4. Clip per-coordinate updates, so no single bad batch moves any weight far.

The guardrail that matters is on the input batch, not the output metric, because the output metric is 40 minutes late and the input check is free. That reordering turns a 5-hour outage into an 86-minute one. The architecture rests on the batch fallback being permanently maintained, kept correct and warm even though it never serves in steady state.

Serving, and the model-size ceiling

Follow one request from arrival to response, then derive the maximum number of parameters the ranking model may have from a deadline set by an external exchange, before quality is discussed.

flowchart TD
    REQ(["Ad request<br/>user · page · slot"]) --> TGT["Targeting retrieval<br/>inverted index over<br/>geo · demo · keyword · audience"]
    TGT --> ELIG["Eligibility<br/>budget · pacing<br/>frequency cap · brand safety"]
    ELIG --> C1["~1,000 candidates"]
    C1 --> LIGHT["Light ranker<br/>sparse LR · 30 kFLOP each<br/>30 MFLOP total"]
    LIGHT --> C2["top 100"]
    C2 --> EMB[("Embedding store — authoritative-copy tier<br/>1.15 GB hybrid table<br/>~4,500 random reads/request")]
    EMB --> HEAVY["Heavy ranker<br/>Wide & Deep · 1.45 M params<br/>2.9 MFLOP × 100 = 291 MFLOP<br/>batched into ONE matrix multiply"]
    HEAVY --> CORR["Calibration layer — where the money is made<br/>+ ln(w) downsample offset<br/>+ per-segment isotonic map"]
    CORR --> AUC{"Auction<br/>eCPM = 1000 · p · bid<br/>vs CPM bids vs floor"}
    AUC --> EXPL{"1.5% exploration<br/>randomize among top 20"}
    EXPL --> SERVE(["1–5 ads + prices"])
    SERVE --> LOG[("Impression log — authoritative copy<br/>+ slot · + explore flag<br/>+ the p that was served")]
    LOG -.->|windowed join with clicks| ONL["Online trainer — no rollback point<br/>FTRL · 15-min checkpoints"]
    ONL -.->|weights| HEAVY

Two things stand out. The funnel narrows before it gets expensive: the cheap model runs on 1,000 candidates, the expensive one on 100. And the calibration layer sits between the ranker and the auction, which is the structural claim of the whole lesson. The impression log is the one authoritative copy every label, calibration fit and backfill derives from, and the only box anything is written to. The online trainer is the one rung you cannot undo: no epoch boundary, so a corrupted hour is absorbed and cannot be un-absorbed, which is what the 15-minute checkpoints bound. Two dotted arrows are asynchronous: nothing on the request path waits for the log-to-trainer loop.

The budget, decomposed

All 100 ms have to be accounted for, so the 20 ms left for scoring is a residual, not a preference:

exchange-imposed auction wall clock                  100 ms
  network RTT, bidder <-> exchange                    -40 ms  -> 60 ms bidder budget
  targeting retrieval + eligibility                   -18 ms
  feature assembly + embedding gather                 -14 ms
  auction, pacing, price, encode                       -8 ms
  ------------------------------------------------------------
  MODEL SCORING BUDGET                                 20 ms

Nobody chose 20 ms. It is what nothing else wanted.

The ceiling

Three facts turn 20 ms into a parameter count: one request runs on one CPU core (splitting a 20 ms unit across cores costs more in coordination than it saves), a modern core sustains ~60 GFLOP/s on a batched 32-bit matrix multiply, and a dense network costs ~2 FLOP per parameter per input (one multiply, one add).

FLOP budget   = 0.020 s × 60e9      = 1.2 GFLOP/request
per candidate = 1.2 GFLOP / 100     = 12 MFLOP
params ceiling = 12 MFLOP / 2       = 6 M parameters

Six million, and not one more. What ships is well under: 780→1024→512→256→1 is 1.45M parameters = 2.91 MFLOP/candidate = 4.85 ms for the batch of 100, a quarter of the budget. By contrast a 100M-parameter model is 333 ms (17× over a hard external deadline) and 1B is 167× over. A 100M model is not “a bit slow”; it is unbuildable here, and model size is decided before quality is discussed (the same shape of argument as the smart-compose latency budget).

The asymmetry: indexed vs traversed

The embedding table is 200× larger than the MLP, yet each request reads only ~288 KB of it (0.025%, the 4,500 rows the 100 candidates need), while every batch traverses 100% of the 5.8 MB of MLP weights. The lookup table may be enormous because it is indexed; the dense network must be tiny because it is traversed. Parameters you index are free; parameters you multiply are not. That is why the answer to “make the model bigger” in ad serving is always “make the embedding table bigger.”

Two implementation notes stand between the 4.85 ms figure and fiction. Batch the 100 candidates into one matrix multiply: scored one at a time, the 5.8 MB of weights streams out of memory 100 times (580 MB at ~100 GB/s = 5.8 ms, more than the arithmetic), making the system memory-bound, not compute-bound, and breaking the 60 GFLOP/s assumption. And compute the user-side representation once, outside the per-candidate loop, since it does not depend on the candidate. The load-bearing assumption is 60 GFLOP/s sustained on one core; a core would have to be 17× faster for a 100M model to fit, which is a different decade of hardware, not a measurement error.

Scale and cost

2.0e10 requests/day -> 231k QPS average, ~700k peak
per request: light 30 MFLOP + heavy 291 MFLOP = 321 MFLOP
peak fleet = 700k × 321 MFLOP = 225 TFLOP/s
per server = 32 cores × 60 GFLOP/s = 1.92 TFLOP/s
  -> 117 servers of pure scoring, ~350 with retrieval + headroom, +40 for the embedding tier

Against $32.9B of revenue, the serving fleet is 0.004% of revenue. A 0.1% revenue improvement is worth $32.9M (24 entire serving fleets) and closing the calibration gap from s_e = 0.25 to 0.10 is worth $2.42B, about 1,800 serving fleets. Cost is not the binding constraint in ad ranking; latency is. The team will spend anything for a tenth of a RIG point and still cannot ship a 100M-parameter model, because the constraint is a 20 ms deadline set by someone else’s exchange.

Failure modes

Every failure here shares one shape: the click log is not a sample of the world. It contains only ads that were shown, only in the slots they occupied, only with the outcomes that had arrived by the time you looked, and only against creatives advertisers optimized toward whatever you rewarded last quarter.

The feedback loop: you only observe clicks on ads you showed

A new ad enters with a prior (the advertiser mean). A pseudocount controls how fast that prior fades: an ad is treated as if it already had m = 700 impressions of the advertiser average, so its own data outweighs the prior only after ~700 impressions. If the prior lands below the winning threshold, the ad never gets an impression, never accumulates evidence, and stays at the prior forever. 61% of new ads reach 700 impressions within 7 days; 24% never reach it, decided forever by a prior they never update. No offline modelling fixes this; the missing thing is data that does not exist.

The fix is exploration, and the right way to argue for it is to price it. Allocate 1.5% of auctions to a randomized winner among the top 20 candidates. By the same Monte-Carlo as the optimizer’s curse, a random pick from the top 20 realizes ~52% of what the argmax of 100 does, so the cost is 1.5% × (1 − 0.52) = 0.73% of revenue ≈ $240M/yr. It earns that back three ways: new ads ramp in 1.2 days instead of 9; new-advertiser 30-day retention rises 4 points, which raises second prices through competition; and the randomized slice is the only unbiased dataset in the system, because its propensity (the probability each ad had of being shown) is known exactly. That makes it ground truth for position-bias estimation and for any off-policy evaluation (estimating how a system you did not deploy would have performed, from logs of the one you did).

Thompson sampling does the same job more cheaply, drawing each ad’s rate from a Beta distribution fitted to its own successes and failures and showing whichever draws highest, so it explores in proportion to genuine uncertainty. Keep a small uniform slice regardless, because its propensities are exact and Thompson’s are not.

Position bias

The log records clicks without recording whether the user looked. Run the same ads in different slots and observed CTR falls off a cliff; dividing each slot’s CTR by slot 1’s estimates how often users even looked:

A logged click is examined AND relevant, and the log records only the product. Train on raw clicks and the model learns that ads placed in slot 1 are good, and the ranker is what placed them there. The loop is self-confirming, and it shows up as COPC by slot of 1.31/0.88/0.71/0.64: a 2× spread, exactly the segment-structured error, entirely manufactured by the training data. Two non-equivalent fixes:

(a) Position as a feature at train time, held constant at serve time. If examination and relevance enter additively in logit space, logit p = f(x) + g(slot), then including slot lets the model isolate examination in g, and serving with slot = 1 for every candidate removes it. Cheap and effective, but only to the extent the additive assumption holds, and it does not fully, because the ranker’s slot assignment is correlated with x.

(b) Inverse propensity weighting (IPW). Weight each impression by 1 / P(examine | slot), so an impression in a slot users look at a fifth of the time counts for five. Estimate the propensities from a randomization experiment. IPW is unbiased but costs variance. The effective sample size (ESS/n = E[w]² / E[w²]) measures how many equally-weighted rows your unequally-weighted rows are worth; on the slot mix here it works out to ~0.756, so you give up about a quarter of your effective sample size in exchange for an unbiased estimate. At 1.8e12 rows that is a trade you take without hesitating, which is why IPW is standard in advertising and rare in small recommenders. What drives the cost is the spread of the weights, not their mean, which is why the deepest slots are usually clipped.

Apply both fixes together and COPC by slot moves from 1.31/0.88/0.71/0.64 to 1.02/0.99/0.97/0.95. A 2.05× spread becomes 1.07×. The load-bearing assumption is that examination and relevance combine additively in log-odds; fix (a) is valid only if it holds, and because it does not fully, (a) and (b) are applied together, not as alternatives.

Delayed conversions, and the bias they inject

Clicks arrive in seconds; conversions do not, and the attribution window truncates the label. pCVR is the predicted conversion rate. Overall, 52% of conversions arrive within 1 h, 79% within 24 h, 94% within 7 d, 99% within 30 d. Train on a 24-hour window and 21% of true positives are labelled negative: they converted, but not yet.

This is exactly the downsampling prior shift with a different cause: there you deleted 95% of negatives on purpose (odds move by w = 0.05); here the world deletes 21% of your positives (odds move by F(24h) = 0.79, so ln(0.79) = -0.236). Same algebra, same one-constant fix, if it were uniform. It is not:

Someone installs a mobile game in ten minutes and finances a car over three weeks. That is a 62-point calibration spread between verticals that compete in the same auctions: travel and auto advertisers systematically lose auctions they should win, to gaming advertisers whose labels merely arrive faster, and the platform loses the difference. This is the single most expensive quiet bug in conversion modelling.

The principled fix models the delay instead of truncating it. Split into a conversion head predicting p (converts eventually, no deadline) and a delay head predicting how long it takes, with cumulative distribution F (so F(24h) = 0.79). Then write the likelihood for a pending impression honestly. It is not a negative, it is “either it never converts or it converts later, and I cannot tell which yet”:

from math import exp, log

def delayed_feedback_nll(p, lam, converted, delay, elapsed):
    """Negative log likelihood for one impression under an exponential delay.
    The whole content is refusing to call a young impression a negative."""
    eps = 1e-12
    if converted:
        return -(log(p + eps) + log(lam + eps) - lam * delay)
    survived = 1.0 - p * (1.0 - exp(-lam * elapsed))   # not (1 - p): only the mass that should have arrived
    return -log(max(survived, eps))

For an impression the model gives p = 0.30: one hour elapsed with F(1h) ≈ 0.02 gives 1 − 0.30·0.02 = 0.994, costing almost nothing (we do not know yet); 30 days elapsed with F(30d) = 0.99 gives 1 − 0.30·0.99 = 0.703, now real evidence against conversion. The cheap alternative is a per-vertical logit offset ln F(window), literally the downsampling correction with w = F(window), applied per segment.

Advertiser-side gaming

The only distortion caused by other people responding to your system. Optimize pCTR and you select for the creative that gets clicked, which is not the creative that is useful, and advertisers build toward what you reward. The top CTR decile of creatives, versus the median, has −41% post-click conversion, +3.1× back-button-within-3s, +2.4× hide rate, and −9 points on 30-day advertiser renewal.

Every defence moves the objective further down the funnel, and every step makes the label sparser, later, and more segment-biased:

Naming the tradeoff is the answer; the usual choice is pCTR · pCVR · value with a quality multiplier and a hide-rate guardrail. Two adjacent problems belong to the same family: invalid traffic (IVT) (bots inflating a publisher’s click rate) must be filtered upstream of training, not only of billing, or the model learns “this publisher is excellent.” And the auto-bidder closes a loop with the advertiser’s own optimizer: it consumes pCVR to set bids, the bids change which impressions are won, and those become next week’s training data. You are modelling a world that reads your output.

Summary of the failure modes

The detection column doubles as a monitoring checklist: a failure with no detector is one you learn about from an advertiser.

Measuring it: offline and online

Offline gate

No single number suffices, because the most popular one is blind to the main failure. PR-AUC is the area under the precision-recall curve; a reliability diagram plots predicted vs observed rate, with a calibrated model on the diagonal.

Two facts about AUC deserve to be raised before anyone asks. Global pooled AUC measures the wrong comparison. Pooled throws every impression into one ranking; grouped (per-auction) computes AUC inside each request and averages. Pooled, the model is rewarded for telling a gaming ad at 9pm from a B2B ad at 9am, a distinction the auction never makes, since it only compares candidates within one request. Here pooled AUC is 0.812 and grouped AUC is 0.594, and only the second is revenue. And AUC cannot see the failure that costs most: it is invariant to every strictly increasing transform of the score, which is exactly what the missing ln(w) offset, the segment biases, and the delayed-label offset are. A perfect-AUC model with a missing ln(w) offset is wrong by 20× on the odds of every impression.

The launch gate is therefore a conjunction: RIG must improve and per-segment COPC dispersion must not regress. RIG alone ships the miscalibrated model; dispersion alone ships the model that learned nothing. The gate depends on offline COPC dispersion predicting the online s_e, which fails in one specific way: a calibration error structured along a dimension not in the fixed grid (a new publisher, an unmodelled device class) is invisible to the metric and fully visible to the auction, which is why the grid must be revisited when traffic composition changes.

Online A/B, and the interference problem

Sizing is easy and that is the trap. At 80% power and 5% significance, detecting a 0.5% relative RPM lift needs ~2.68M users per arm (16σ²/δ²), about half a percent of a 500M-DAU platform, powered within a day. So the danger is peeking, not power: checking hourly and stopping when it looks significant takes the false-positive rate from 5% well past 20% (multiple testing). Fix the horizon in advance or use a sequential test.

The real difficulty is interference: the two arms are not independent. Randomizing users does not isolate advertisers, because budget is shared across arms: treatment wins more auctions for advertiser X, X’s budget drains faster, X goes ineligible in both arms, and control looks worse for losing access to the same ads. So the measured effect partly reflects a resource transfer between arms, and it does not replicate at 100% rollout where there is no control arm to take budget from.

A switchback randomizes time, not users (the whole marketplace runs treatment for an interval, then control) because there is no way to give one user a different floor than another. A budget-split removes the budget channel but not marketplace effects (treatment changing what competing bids look like), which is why the switchback row exists.

Alternatives considered and rejected

Conclusion

• The output is a price input, not a rank. Calibration is a correctness requirement, and log loss (a strictly proper scoring rule) is the objective, not a ranking loss.
• Uniform bias cancels in a second-price auction; the real failures are where a modeled number meets a non-modeled one (the fixed floor, CPM competitors, and the auto-bidder) combined with segment-structured error amplified by taking a maximum over 100 candidates (the optimizer’s curse, worth ~$2.42B/yr between s_e = 0.25 and 0.10).
• AUC is structurally blind to all of this. Report RIG and gate on per-segment COPC dispersion, never on global COPC or pooled AUC.
• Two prior-shift biases are injected on purpose and fixed by one constant in log-odds each: negative downsampling (ln(w) = -2.9957) and, per segment, delayed-conversion truncation (ln F(window)). A per-segment isotonic map cleans up the rest.
• Model size is set by latency, not quality: a 20 ms budget over 100 candidates at ~60 GFLOP/s per core caps the dense network at ~6M parameters. Indexed embeddings can be gigabytes; the traversed MLP must be megabytes.
• Every remaining failure comes from the click log not being a sample of the world (only shown ads, only in their slots, only with outcomes that had arrived, only against gamed creatives) and each has a matching detector and control.

One line to remember: the model’s output is a price, not a rank, so anything that leaves the order intact but shifts the number (downsampling, delayed labels, segment drift) is invisible to AUC and costs real money until you calibrate per segment.

Further reading

• McMahan et al., “Ad Click Prediction: a View from the Trenches” (KDD 2013): FTRL-Proximal, per-coordinate rates, and hard-won lessons on calibration and feature management.
• Cheng et al., “Wide & Deep Learning for Recommender Systems” (2016): the memorize-and-generalize architecture used by the heavy ranker.
• Rendle, “Factorization Machines” (ICDM 2010): the model and its linear-time closed form.
• Chapelle, “Modeling Delayed Feedback in Display Advertising” (KDD 2014): the delayed-conversion likelihood with a delay head.
• Wang et al., “DCN V2: Improved Deep & Cross Network” (2020): learned bounded-degree crosses.
• Smith & Winkler, “The Optimizer’s Curse” (Management Science 2006): why the winner of a max over noisy estimates disappoints.

Next: the similar-items chapter, where the score is neither a rank nor a price but a retrieval key, “similar” turns out to mean substitutable, and the hard constraints are dates and availability, not milliseconds.