08 — Ad Click Prediction

“Predict the probability a user clicks an ad, so we can run the auction.”

Every other ranking system in this track cares about the order of its scores. This one cares about the scores themselves, because they get multiplied by money.

By the end you should be able to:

• Name every model in an ad serving stack and say what each one eats and emits.
• Derive the maximum size of the ranking model from a latency budget someone else set, before quality is even discussed.
• Explain precisely why a probability that is systematically 25% too high breaks the auction rather than merely the metric — and why the obvious version of that argument is wrong.
• Correct the two biases that a real training pipeline injects on purpose, one from downsampling and one from delayed labels.

The worked example is a platform serving 2.0e10 ad requests a day against $32.9B of annual revenue.

The one idea

Calibration here is a correctness requirement, not metric hygiene.

In a search ranker, a miscalibrated score is an aesthetic complaint: reorder nothing, lose nothing. In an auction, the predicted probability is multiplied by a bid to make a price, compared against a fixed reserve, and fed to a budget controller.

Get it wrong and three things happen at once. The wrong ad shows. The publisher is underpaid. An advertiser’s campaign is throttled. Meanwhile every rank-based metric on your dashboard stays exactly where it was, AUC very much included.

AUC — the area under the receiver-operating-characteristic curve — is the probability that a randomly chosen clicked ad scores above a randomly chosen unclicked one. It is the industry’s default quality number, and Offline metrics explains why it is the wrong one here.

Sections Why calibration is non negotiable here, Extreme imbalance negative downsampling and the correction it forces and Delayed conversions and the bias they inject are three different causes of that same failure.

Most ranking problems consume the order of your scores. This one consumes the score itself: the predicted probability is multiplied by an advertiser’s bid to produce an expected value, and that value is compared against other bids, against a reserve price, and against a budget forecast. A number that only has to be in the right order is a rank; a number that gets multiplied is a price. Everything distinctive about this chapter follows from that.

First thing to say: “The output of this model is not a ranking, it is a price input. So calibration is a first-class requirement rather than a post-processing step, and AUC — which is invariant to any monotone transform of the score — cannot see the failure that costs the most money. Separately, the auction has a ~20 ms scoring budget for a hundred candidates, which sets the model-size ceiling before we have discussed architecture.”

Notation you need before anything else

Four pieces of notation appear on nearly every page below. Get them straight now and the rest of the chapter reads as arithmetic instead of as symbols.

Scientific notation. 2.0e10 means 2.0 × 10^10, or 20,000,000,000. 5e-2 means 0.05. The chapter uses it because the volumes here span nine orders of magnitude and writing the zeros out is unreadable.

Odds. If the probability of a click is p, the odds are p / (1 - p). A probability of 0.01 is odds of 0.01 / 0.99 = 0.0101, or roughly “1 click per 99 non-clicks.” Odds run from 0 to infinity, while probability is squeezed into [0, 1].

Log-odds, also called the logit. logit(p) = ln(p / (1 - p)), the natural logarithm of the odds. This is the scale a logistic model actually works in: the raw number a logistic regression or a neural network emits is a logit, and you convert it to a probability with the sigmoid function p = 1 / (1 + e^-z). The two are exact inverses.

Why it matters here: several distortions in this chapter multiply the odds by a constant. Multiplying odds by a constant is adding a constant in log-odds, because ln(a · b) = ln(a) + ln(b). That is why every correction below turns out to be “add one number to every score” rather than anything shape-changing.

p = 0.01000  ->  odds = 0.01000/0.99000 = 0.01010  ->  logit = ln(0.01010) = -4.5951
p = 0.16807  ->  odds = 0.16807/0.83193 = 0.20202  ->  logit = ln(0.20202) = -1.5994

multiply the odds by 0.05:   0.20202 x 0.05  =  0.01010
in log-odds:                 -1.5994 + ln(0.05)  =  -1.5994 - 2.9957  =  -4.5951
                                                      ^ same answer, one addition

The x in 1000 x 0.006 x 1.20 is a multiplication sign. Code blocks in this chapter use x and · interchangeably for multiply.

Two ideas borrowed from elsewhere in this repo

Both do enough work below that they are restated here in full, so nothing in this chapter requires reading another one first.

Calibration means that among all the times a model says 0.02, about two in a hundred really do click. It is a completely separate property from ranking: a model can order every candidate perfectly and still be systematically double on every score. Calibration what it means and when it matters develops it in general; this chapter is where it becomes money.

Prior shift is what happens when the base rate in your training data is not the base rate in production — because you threw away most of the negatives, or because a label had not arrived yet. The correction turns out to be a single constant added to every score, provided you work in log-odds; The correction derived derives it from scratch and Resampling and the calibration it breaks gives the general version.

The models in this design, and how you know each one works

An ad serving stack is not one click model. There are two rankers in a cascade, a calibration layer that stands between the ranker and the money, and four smaller models that exist to correct biases the logs cannot avoid.

Here is the roster. Each row answers the same five questions: what the model is, what goes in and comes out, where its training labels come from, the one number that says it works, and whether it runs on the live request path (online) or on a schedule over logs (offline).

The table is a map, not a lesson — every row is built up properly in a later section, and the section numbers are in the cells. On a first read, only the shape matters: one of these models produces the score, one of them fixes the score, and four of them exist purely to undo damage the logging did.

Two things in that table are worth reading twice. The heavy ranker’s output is not the system’s output — the calibration layer stands between them, and every argument in this chapter about money is an argument about that layer rather than about the network. And the last four rows are not accuracy improvements; they are bias corrections. Each one exists because the click log is not a sample of the world: it only contains ads that were shown, in slots users may not have looked at, with conversions that may not have arrived yet.

1. Framing

Everything the rest of the chapter derives — the latency ceiling, the dollar figures, the bias corrections — traces back to a handful of numbers fixed here: the input, the output, the volumes, and the five clarifying questions whose answers change the design rather than decorate it.

The vocabulary of ad serving

Seven abbreviations run through the whole chapter. Learn them here and the tables below stop being alphabet soup.

eCPM is the one that does the work, so be sure of it. A CPM advertiser already bids in eCPM — they named a price per thousand impressions, so their bid is their eCPM. A CPC advertiser bids per click, so to compare them you have to guess how many clicks a thousand impressions will produce. That guess is the model’s output:

CPC advertiser bids $1.20 per click, model predicts 1.8 % click rate

1,000 impressions  x  1.8 % click rate      =  18 expected clicks
18 clicks          x  $1.20 per click       =  $21.60 expected revenue
                                               per 1,000 impressions

so:  eCPM  =  1000 x p x bid  =  1000 x 0.018 x 1.20  =  $21.60

The model’s error goes straight into that dollar figure. That is the whole chapter in one line.

The problem, in numbers

This table fixes the inputs, outputs and volumes every later derivation uses. The two rows to remember are Latency, which The ceiling derived turns into a hard parameter ceiling, and Revenue, which turns every percentage in the chapter into dollars.

Two of those rows are multiplications rather than given facts, so do them on paper once:

volume    500e6 users x 40 impressions each      =  2.0e10 impressions/day
          2.0e10 / 86,400 seconds in a day       =  231,000 QPS average
          peak is ~3x average on a diurnal curve =  ~700,000 QPS peak

revenue   2.0e10 impressions x 1.0 % click rate  =  2.0e8 clicks/day
          2.0e8 clicks x $0.45 per click         =  $90e6  =  $90M/day
          $90M x 365 days                        =  $32.9B/yr

Hold on to $32.9B/yr. Every “this costs 5.1% of revenue” claim below is that number times a percentage.

The clarifying questions that change the design

An interviewer expects you to ask a handful of questions before designing. These are the ones whose answers change the architecture rather than decorate it — the last column says what breaks if the answer flips.

Assumptions this chapter runs on, and the two that are load-bearing. Most of the figures above are stated by the interviewer and can move by a factor without changing an argument. Two cannot:

• Load-bearing: CPC and CPM advertisers compete in the same auction. This single answer is what makes calibration a correctness requirement rather than a preference. If every bidder went through the model, a uniform bias would cancel exactly and Why calibration is non negotiable here would be a much shorter section. Because some bids are known exactly and some are predicted, every mixed auction compares a modeled number against a known one, and there is nothing for the error to cancel against.
• Load-bearing: the 20 ms scoring budget, and that it is imposed from outside. It is a slice of an exchange’s 100 ms wall clock, so it is not negotiable by the team that owns the model, and it sets a hard parameter ceiling in The ceiling derived before quality is discussed at all.
• Not load-bearing, though the numbers are large: the 1% base rate, the $0.45 average cost per click, and the 500M daily active users. They scale the revenue figures and therefore the dollar value of every improvement, but no ranking or design decision flips if they are wrong by 30%.

2. The ML objective

The loss function is unusually consequential here: it is the one place where “it is a ranking problem” gives the wrong answer. Just as consequential is exactly which parts of the auction consume the score, because that is what the rest of the chapter argues about.

2.1 The loss: log loss, and why not a ranking loss

The setup is pointwise binary classification: score each (request, candidate) pair on its own, with a label of 1 for click and 0 for no click, and train with log loss. “Pointwise” means one row per candidate, judged in isolation — as opposed to pairwise, which trains on “A should beat B.”

L  =  - (1/N) sum_i [ y_i log p_i  +  (1 - y_i) log(1 - p_i) ]

Read that formula one term at a time. y_i is the label, 0 or 1. p_i is the model’s predicted probability. When y_i = 1 the second bracket term vanishes and the loss is -log p_i, which is 0 if you said 1.0 and grows without bound as you approach 0. When y_i = 0 the first term vanishes and you are penalized by -log(1 - p_i) instead. Averaged over N rows, it is the penalty for confident wrongness.

The loss choice is not a default, and here is why it is forced.

Log loss is a strictly proper scoring rule, so its unique minimizer is the true conditional probability (Proper scoring rules log loss and brier).

A proper scoring rule is a loss with one property: the way to score best is to report what you actually believe. Any other answer scores worse in expectation. Strictly proper means the honest answer is the only answer that achieves the minimum — no second-best tie.

That is exactly the guarantee an auction needs, because the auction consumes the number and not the order.

A pairwise ranking loss has no such property. Its minimizer is any monotone transform of the truth — double every probability, square it, take its log; the ordering is unchanged, so the ranking loss is unchanged and equally minimized. That is precisely the family of transforms that costs money here. That single line disqualifies LambdaRank-style objectives for this problem, and it is worth saying before anyone proposes one.

2.2 What the auction does with p

Two definitions before the formulas. A reserve price or floor is the minimum eCPM a publisher will accept for the slot; below it, they show nothing. A generalized second price (GSP) auction is one where the winner pays what it would have taken to beat the runner-up rather than what they actually bid — which is why the runner-up’s eCPM, written eCPM_2, appears in the price formula.

eCPM_i  =  1000 · p_i · bid_i                 for a CPC advertiser
eCPM_j  =  bid_j                              for a CPM advertiser (no model)

show if  max_i eCPM_i  >=  floor
generalized second price, winner pays per click:

           price_1  =  eCPM_2 / (1000 · p_1)

The division by 1000 · p_1 in the last line is a unit conversion, and it is worth doing once with numbers rather than staring at it. eCPM_2 is a price per thousand impressions. A CPC advertiser is billed per click. So you convert, using the worked auction of One row populated:

runner-up eCPM_2                        =  $21.60 per 1,000 impressions
winner's calibrated click rate p_1      =  0.02092  (2.092 %)

1,000 impressions x 0.02092             =  20.92 expected clicks
$21.60 spread over 20.92 clicks         =  $1.03 per click

as a formula:  21.60 / (1000 x 0.02092) =  $1.03     <- price_1

Three separate consumers of the number, and they are the three the whole chapter keeps returning to:

1. The ranking picks the largest eCPM.
2. The pricing divides by p.
3. The floor comparison tests eCPM against an absolute number the model has no part in.

Only the first is invariant to a monotone rescaling of p. A bias a ranker would never notice lands directly on the other two.

3. Why calibration is non-negotiable here

One claim carries the whole chapter: a miscalibrated click model breaks the auction itself, not merely the metric that measures the model. The argument runs in four steps — the naive version of the claim and why it is wrong, the three places the bias genuinely survives, what heterogeneous error actually costs in dollars, and the one instrument that detects it in production. Start with the case against yourself, because the version of this argument most people give does not hold.

3.1 The naive argument, and why it fails

The standard answer is: “Miscalibration breaks the auction because the wrong ad wins.” Check that claim rather than repeating it, because the real argument only has room once it is out of the way.

Suppose every predicted CTR is inflated by the same factor k — say k = 1.25, so every prediction is 25% too high. Multiplying every score by the same number cannot change their order, so the ranking is untouched. Work the price through as well:

eCPM_i' = 1000 · k · p_i · bid_i       every candidate scales by k
argmax  unchanged
price_1 = eCPM_2' / (1000 · p_1')  =  (k · eCPM_2) / (1000 · k · p_1)  =  price_1

That last line is where the argument dies. The GSP price is a ratio: the runner-up’s eCPM on top, the winner’s predicted rate underneath. Inflating every prediction by k multiplies the top by k and the bottom by k, and they cancel. Substitute to be sure:

honest:     p_1 = 0.020,  eCPM_2 = $21.60
            price = 21.60 / (1000 x 0.020)          =  $1.080

inflated 1.25x:
            p_1' = 0.025,  eCPM_2' = 21.60 x 1.25   =  $27.00
            price = 27.00 / (1000 x 0.025)          =  $1.080   <- identical

A uniform multiplicative bias cancels exactly in a second-price auction between two ads scored by the same model. Say this out loud in the interview. It is the mark of someone who has actually worked the algebra, and it forces you to find the places where the bias genuinely does not cancel — which is where the real argument is.

3.2 Three places the bias does not cancel

The cancellation in The naive argument and why it fails worked only because both sides of every comparison came out of the same model. So the failures are exactly the three comparisons where that is false — against a fixed number set by the publisher, against a bid that never touched the model, and against a controller that reads the probability itself rather than comparing it to anything.

(a) The reserve price is an absolute number, and the model is not in it

The floor is a dollar figure the publisher typed into a config file. It does not move when your model moves. So a comparison against it has nothing to cancel against.

Follow one ad that should not have been shown:

publisher floor                    $8.00 eCPM
ad D: true CTR 0.60 %, bid $1.20   true eCPM  =  1000 x 0.0060 x 1.20  =  $7.20
                                                          -> should NOT show

model overpredicts by 1.25x        pred eCPM  =  1000 x 0.0075 x 1.20  =  $9.00
                                                          -> shows
publisher realizes $7.20 on a slot they priced at $8.00

Step by step: the ad’s honest worth is $7.20 per thousand impressions, which is below the $8.00 floor, so it should lose the slot. Inflating the CTR by 1.25x turns 0.60% into 0.75%, which turns $7.20 into $9.00, which clears the floor. The ad shows. The publisher was promised $9.00 of value and gets $7.20 — an 80-cent shortfall on a slot they had explicitly priced.

Roughly 6% of auctions sit within 15% of the floor, so this is a persistent leak concentrated exactly where the publisher is most price-sensitive.

(b) CPM advertisers do not go through the model at all

This is the cleanest version of the argument, so it is the one to lead with. A CPM advertiser states a price per thousand impressions directly. No model touches it. So every auction that mixes CPC and CPM bidders compares a predicted number against a known one.

The block below runs two ads through that comparison, one where the model under-predicts and one where it over-predicts. Watch that the publisher loses money in both:

Ad A: CPC, bid $1.20/click, TRUE CTR 1.80 %
      true eCPM  =  1000 x 0.0180 x 1.20  =  $21.60
Ad B: CPM, bid $19.00 per 1,000 impressions   (known exactly)

truth: A should win; the publisher earns $21.60 per 1,000 impressions

model UNDERpredicts CTR by 0.85x:
      pred eCPM_A  =  1000 x 0.0153 x 1.20  =  $18.36
      B wins.  Publisher earns $19.00.
      loss  =  $2.60 / 1,000 impressions  =  12.0 %

model OVERpredicts by 1.25x on ad C (true CTR 1.20 %, bid $1.30):
      true eCPM_C  =  $15.60,  pred eCPM_C  =  $19.50
      C beats a $17.00 CPM ad. Publisher expected $19.50, realizes $15.60.
      loss  =  $1.40 / 1,000 impressions  =  8.2 %  vs taking the CPM ad

The two loss figures are worth reconstructing, because neither is obvious:

under-prediction case:
   what the publisher could have had  =  $21.60   (ad A, honestly priced)
   what the publisher got             =  $19.00   (ad B won instead)
   shortfall                          =  $2.60
   as a share of what was available   =  2.60 / 21.60  =  12.0 %

over-prediction case:
   what the publisher could have had  =  $17.00   (the CPM ad it displaced)
   what the publisher actually got    =  $15.60   (ad C's real worth)
   shortfall                          =  $1.40
   as a share of what was available   =  1.40 / 17.00  =  8.2 %

Every mixed auction is a comparison between a modeled quantity and a known one, and there is nothing for the bias to cancel against.

Notice the damage runs in both directions and both directions cost the publisher. Under-prediction hands the slot to a CPM bid worth less than the CPC ad it displaced. Over-prediction takes a slot away from a CPM bid that was genuinely worth more. There is no direction of error that is safe.

(c) The auto-bidder consumes the probability directly

Two more services to define, because both read p as a raw quantity.

Target-CPA auto-bidding is a service where the advertiser names what a conversion is worth to them and the platform sets each bid on their behalf. It sets bid = target_CPA · pCVR, where pCVR is the predicted conversion rate.

Pacing is the controller that spreads a daily budget evenly through the day instead of spending it all by 9am. It forecasts spend as sum of p · price.

Neither compares the probability to another model’s output. Both use it as a number in its own right, so neither has anything to cancel against.

Inflate p by 1.25x and the pacer believes the budget will exhaust 25% early, so it throttles. The advertiser under-delivers. The failure surfaces as a sales escalation, not as anything on the machine learning dashboard.

3.3 The real cost: heterogeneous error and the optimizer’s curse

What does calibration error cost in dollars? The number that comes out justifies the whole chapter’s emphasis, and reaching it takes four steps: why the error is not uniform, why taking a maximum is dangerous, the model, and the dollar table.

Real error is segment-structured, not uniform

A segment here is a cell of the traffic defined by things like the advertiser’s industry (the vertical), the device class, the publisher, and the time of day (the daypart). “Mobile · gaming · slot 1 · 9pm” is one segment.

Real calibration error is segment-structured: any one cell can be systematically over- or under-predicted while the others are fine. Uniform bias — the kind that cancelled in The naive argument and why it fails — is a fiction.

An auction takes a maximum, and maxima are biased

An auction does not average over candidates. It takes the maximum over them, 100 at a time.

That is a problem, and the reason is not obvious. Suppose every candidate’s score is its true worth plus some random error. When you pick the highest score, you are not only picking a genuinely good candidate — you are also picking, disproportionately, a candidate whose error happened to be positive. High-error-in-your-favour and high-true-worth both push a candidate to the top, and you cannot tell them apart from the score alone.

That is the optimizer’s curse: whatever you pick as best is, on average, worse than your estimate of it said, and the more candidates you choose among, the larger that gap. Choosing among 100 candidates makes it much worse than choosing among 2.

The model

Write the true worth of a candidate on a log scale, because eCPMs span orders of magnitude and multiplicative spread is easier to reason about than additive.

• v_i is candidate i’s log true worth, drawn from N(0, s_v^2) — a normal (bell-curve) distribution with mean 0 and standard deviation s_v.
• m_i = v_i + e_i is the model’s score: the truth plus an error e_i ~ N(0, s_e^2).
• s_e is therefore the size of the model’s calibration error, measured in log-odds — the single number this whole subsection is about.

draw n log-values     v_i  ~  N(0, s_v^2)                candidate true worth
model score           m_i  =  v_i + e_i,   e_i ~ N(0, s_e^2)
auction shows         argmax_i m_i          the model's opinion of the max, not v's

regression to mean    E[v | m]  =  m · r^2,   r  =  s_v / sqrt(s_v^2 + s_e^2)

revenue  =  E[ exp(v_selected) ]                     realized worth of the winner
optimal  =  E[ exp(v_best) ],   v_best = max_i v_i   worth of the true best

Two lines in that block deserve names.

Regression to the mean is the line E[v | m] = m · r^2. Given a score m, the true value is expected to be closer to average than the score is — you shrink the score toward 0 by a factor r^2. The quantity r measures how much of the score is signal rather than error. Substitute the table’s numbers once:

s_v = 0.6 (spread of true worth),  s_e = 0.25 (model error)

r    =  0.6 / sqrt(0.6^2 + 0.25^2)
     =  0.6 / sqrt(0.36 + 0.0625)
     =  0.6 / sqrt(0.4225)
     =  0.6 / 0.65
     =  0.9231

r^2  =  0.852   ->  a score of 1.0 is really worth about 0.85

The bottom two lines are the numerator and denominator of the answer. revenue is what the auction actually realizes — the true worth of whichever candidate the model thought was best. optimal is what it could have realized had it picked the genuinely best candidate. Their ratio is the “Revenue vs optimal” column below.

Why the answer is simulated rather than derived

That ratio has no honest closed form, and both shortcuts people reach for inflate it.

Shortcut one, the max-of-n formula. E[max of n] ≈ s·sqrt(2 ln n) overstates the maximum by about a fifth at these sizes. For n = 100: sqrt(2 · ln 100) = sqrt(9.21) = 3.03, but a simulated maximum of 100 standard normal draws sits at 2.51 standard deviations, not 3.03.

Shortcut two, exponentiating the log-gap. Worth is exp(v), and the mean of an exponential is not the exponential of the mean. That is Jensen’s inequality: for a curved function like exp, averaging first and then applying the function gives a different answer than applying it first and then averaging.

So simulate the whole thing by Monte-Carlo — draw many random scenarios, compute the outcome in each, and average — rather than trusting a formula.

The dollar table

The simulation below uses n = 100 candidates (Framing: 100 reach the heavy model) and s_v = 0.6. That s_v means one standard deviation of candidate eCPM is a 1.8x spread, since exp(0.6) = 1.82.

Read the table left to right: pick a model error, see how much of the theoretically available revenue the auction actually captures, and convert the shortfall to dollars against $32.9B.

The last column is one multiplication. At s_e = 0.25 the auction captures 91.0%, so it leaves 9.0% on the table, and 0.090 × $32.9B = $2.97B.

Halving the log-odds error from 0.25 to 0.10 recovers 7.4 points of revenue — 98.4% − 91.0% = 7.4%, and 0.074 × $32.9B = $2.42B a year on this traffic.

And now the punchline:

AUC averages over all pairs; the auction consumes only the maximum of 100. Those are different statistics, and only one of them pays. A monotone rescaling that leaves AUC bit-for-bit identical moves the table above by a full column.

What you can actually do about s_e

Two properties of s_e decide the shape of the fix, and they are the reason The calibration map fit per segment exists in the form it does.

First, s_e has two components and they are not equally expensive to remove. Refinement error is genuine ignorance about which impressions convert; the only cures are better features or more data. Reliability error is the score being systematically mis-scaled; it can be removed by fitting a one-dimensional map on held-out data, which is cheap. The calibration layer attacks the second.

Second, and worse: systematic error is more expensive than random error. When the error is independent from impression to impression, the regression-to-the-mean term above at least averages the damage out over many auctions. When it is systematic per segment, that protection disappears entirely — an over-predicted segment wins auctions persistently rather than by luck, so its losses accumulate in one direction instead of cancelling. That is what makes segment-structured error the expensive kind, and it is why the fix has to be per-segment too.

3.4 The production instrument: COPC, per segment

One number belongs on the dashboard — and its global version is worse than useless.

COPC stands for clicks over predicted clicks:

COPC  =  observed clicks  /  sum of predicted probabilities

over some slice of traffic. If you showed 10,000 impressions whose predicted probabilities add up to 100 expected clicks and you observed 100 clicks, COPC is 100 / 100 = 1.00 and the model is calibrated on that slice.

• COPC = 1.00 — calibrated.
• COPC > 1.00 — more clicks happened than the model expected, so the model is under-predicting.
• COPC < 1.00 — the model is over-predicting.

Global COPC is nearly useless, because the errors cancel. The table below shows why: five segments, wildly miscalibrated in opposite directions, adding up to a global number that looks perfect. The column to watch is the last one — the signed contributions cancel almost exactly.

segment              share of predicted   COPC     contribution to global
mobile · gaming            32 %           1.34            +0.109
desktop · retail           24 %           0.71            -0.070
mobile · retail            21 %           1.09            +0.019
desktop · news             14 %           0.83            -0.024
other                       9 %           0.66            -0.031
                                                  ---------------------
                                global COPC  =   1.003        <- "well calibrated"

Each contribution is share × (COPC − 1), which is how far that segment pulls the global number away from 1:

mobile · gaming    0.32 x (1.34 - 1)  =  0.32 x  0.34  =  +0.109
desktop · retail   0.24 x (0.71 - 1)  =  0.24 x -0.29  =  -0.070
mobile · retail    0.21 x (1.09 - 1)  =  0.21 x  0.09  =  +0.019
desktop · news     0.14 x (0.83 - 1)  =  0.14 x -0.17  =  -0.024
other              0.09 x (0.66 - 1)  =  0.09 x -0.34  =  -0.031
                                                          ------
                                                          +0.003   ->  COPC 1.003

The weights are shares of predicted clicks, not of impressions. That is not a stylistic choice: COPC = sum(obs) / sum(pred) is a ratio of totals, so each segment enters the global number in proportion to how much it contributes to the denominator.

A globally perfect COPC of 1.003 is hiding a 2x spread — 1.34 down to 0.66 — across the segments that actually compete in auctions with each other.

Two operating rules follow. Alert on per-segment COPC against a fixed grid (vertical × device × slot × hour bucket), never on the global number. And treat the dispersion of segment COPC — how widely the per-cell values are scattered, measured as their standard deviation — as the headline metric, because it is a direct estimate of the s_e in The real cost heterogeneous error and the optimizers curse that the dollar table is priced against.

Assumptions in this section, and the load-bearing one. The dollar table rests on a simulation with three inputs: 100 candidates per auction, a spread of candidate value s_v = 0.6, and log-normal worth. The load-bearing one is s_v = 0.6, the assumed spread of true candidate value, because the cost of the optimizer’s curse is a comparison between how much candidates differ and how much the model errs — halve s_v and the same s_e costs far more, double it and the same error costs less. The candidate count matters much less than it looks, since the curse grows only with the logarithm of n. What is not an assumption, and is the section’s actual claim, is that the error is segment-structured rather than random: that is measured, in the COPC table above, and it is why the fix in The calibration map fit per segment is per-segment rather than global.

4. Data and labels

A model is only as honest as its training rows, so pin down exactly what one row is and where its label comes from. The label here is unusually good — free, instant, and produced twenty billion times a day — which makes it easy to miss that it is also biased in three specific ways, every one of which costs a later section to repair.

An attribution window is the period after showing an ad during which a subsequent action is credited to it. Outside the window, the same click or purchase counts as unrelated.

Here is one training row, end to end:

The volume row is just the base rate applied: 2.0e10 impressions × 1.0% = 2.0e8 clicks. So 99 rows in 100 are negatives, which is what Extreme imbalance negative downsampling and the correction it forces is about.

Three properties of this label

Each one is repaired in a different later section, and together they drive most of the second half of the chapter.

1. It is cheap, immediate and enormous. Nobody labels anything by hand; the user does it, twenty billion times a day, for free. Unlike Data and labels three targets and a nine day delay there is no delay for clicks. For conversions there is, and it is worse (Delayed conversions and the bias they inject).

2. It is conditioned on having been shown. Only shown impressions produce rows at all. So the training distribution is not a sample of the world — it is a snapshot of what the current ranker already believes (The feedback loop you only observe clicks on ads you showed).

3. It is contaminated by placement. A user must look at an ad before they can click it. So a raw click measures examination times relevance, and the log records only the product, never the two factors separately (Position bias).

Joining impressions to clicks

This is a real system, not a footnote. An impression at time t and a click at t + 12 s arrive from separate services, so the join has to wait a bounded interval before it is allowed to declare an impression un-clicked.

That bound is called a watermark — a timestamp the stream processor treats as “everything older than this has certainly arrived.” Set it too short and you drop late clicks; set it too long and your training data is hours stale. Choosing it trades latency against completeness.

Two failure modes matter, and note that they push the same way. Duplicate impressions inflate the denominator. Dropped clicks shrink the numerator. Both bias measured CTR down, and therefore bias the model down.

Reconcile the stream join against a daily batch recomputation and alert above 0.2% divergence. A silent 1% click loss is indistinguishable from a 1% CTR decline, and it will be debugged as a product problem for a week before anyone checks the pipeline.

4.1 Train/serve skew, and point-in-time correctness

One bug produces more silent calibration damage than everything in Failure modes put together, and it belongs here rather than there because it is a property of how a row is built, not of what the world did to it.

Train/serve skew is any difference between how a feature is computed when the model is trained and how the same feature is computed when the model is served. Point-in-time correctness is the discipline that prevents it: every feature on a training row must be computed from only the information that was actually available at the instant that impression was served, never from the complete history the offline job happens to have lying in front of it. The two are the same requirement stated from opposite ends — skew is the failure, point-in-time correctness is the invariant.

Both bite here, hard, because this design puts counter features on the request path. A counter feature is a running total: the ad-level shrinkage prior of The feedback loop you only observe clicks on ads you showed is an ad’s own impressions and clicks; the user-history features of Do not embed the user id are click counts and recency over 1, 7 and 30 days.

A running total is exactly the kind of feature a training job computes wrong by default. The trace below follows one counter, for one ad, at serve time and then at train time the next day. The number to watch is the click count: it should be identical and it is not.

serve time, 09:14:02   ad 41882113 counter reads a live snapshot that is
                       ~90 s behind the click stream        ->  1,180 clicks

train time, next day   the offline job groups the click table by ad_id
                       over "everything before 09:14:02"    ->  1,206 clicks
                          (the 26 clicks that landed in the 90 s lag, plus
                           late-arriving rows the watermark had not admitted)

the model is fitted on a counter no serving process can ever produce

That 26-click gap is not noise and it does not average out. Three things make it dangerous:

• It is systematically upward at train time. Offline always has more history than online did. The error only ever runs one way.
• It is largest on exactly the ads whose counters are moving fastest. A dormant ad’s counter barely changes in 90 seconds; a viral one’s does.
• It varies by vertical, because ad velocity does. So it is not a constant the The correction derived offset could absorb — it lands squarely in the segment-structured s_e of The real cost heterogeneous error and the optimizers curse, which is the expensive kind.

The consequence is that the model learns to trust a counter that is always a little more complete than the one it will actually be handed, so it under-uses the feature at serve time, and it under-uses it unevenly.

Three controls, and none is optional:

1. Log the feature vector that was actually served, alongside the p the serving diagram already logs (Serving and the model size ceiling derived from the auction budget), and train on that rather than recomputing it. This makes point-in-time correctness a property of the log rather than of a query someone has to remember to write correctly.
2. Where a feature genuinely must be recomputed, compute it as of the impression timestamp minus the serving pipeline’s own lag — the 90 seconds above. Treat that lag as a number you measure, not one you assume.
3. Alert on the distribution of every feature, serve-side against train-side. That comparison is the only detector that fires before the revenue does. It is the same argument as The stability side and the incident it prevents’s input-batch guardrail, applied to the feature rather than to the batch.

5. Extreme imbalance, negative downsampling, and the correction it forces

The single most common production bug in ad ranking is one you introduce deliberately. Throwing away most of the negative rows is the right call for compute reasons, it costs almost nothing statistically, and it silently multiplies every predicted probability by twenty in the odds unless you undo it. What follows is the correction, derived; what it does and does not fix; and the three ways teams get it wrong.

CTR is ~1%, so 99% of rows are negatives.

That is an imbalance you should not fix for statistical reasons. Log loss is proper, so it already wants the true probability. The consumer is a probability, not a class label. And there is no threshold to move, because nothing here says “click” or “no click” (Imbalance is usually a threshold problem wearing a data problems clothes).

You downsample negatives for exactly one reason: compute.

The scheme is: keep every positive row, and keep each negative row with probability w = 0.05 — a coin flip that comes up “keep” one time in twenty. w is called the sampling rate.

keep all positives;  sample negatives at rate w = 0.05

pi   =  0.01                            true prevalence
pi'  =  0.01 / (0.01 + 0.05 x 0.99)  =  0.01 / 0.0595  =  0.16807
rows =  0.0595 N                     ->  16.8x smaller training set

Unpack pi' — the base rate inside the sampled data — because everything downstream depends on it. Start with 1,000,000 rows:

positives     1,000,000 x 1 %      =    10,000   all kept        ->  10,000
negatives     1,000,000 x 99 %     =   990,000   keep 5 % of them ->  49,500
                                                 ----------------------------
sampled rows                                                        59,500

pi' = 10,000 / 59,500 = 0.16807     the base rate the model now sees
59,500 / 1,000,000 = 0.0595         so the data is 1/0.0595 = 16.8x smaller

The base rate went from 1% to 16.8%. The model will learn to predict around 16.8%, not around 1%, and nothing in the loss function knows that is wrong. That is the bug this section is about.

5.1 The correction, derived

The fix takes three lines of algebra: it is the The calibration break derived prior shift, specialized to this case.

Three symbols to fix before the algebra:

• pi is the base rate — the overall probability of a click, 1% here. pi' is the base rate in the sampled data, 16.8%.
• f_1(x) is how the features are distributed among rows that clicked.
• f_0(x) is how they are distributed among rows that did not.

The key observation is what downsampling does and does not touch. It throws away negatives only, and it throws them away at random, which means:

• f_1(x) is untouched — you kept every positive.
• f_0(x) keeps its shape — dropping negatives uniformly at random does not change what a typical negative looks like — but its total mass is multiplied by w.

So the ratio f_1/f_0 is unchanged, and only the mixing proportion moves. Write the odds in both worlds:

o'  =  [ pi' / (1 - pi') ] · f_1/f_0        sampled world
o   =  [ pi  / (1 - pi ) ] · f_1/f_0        true world
                                            ->   o  =  w · o'

logit(p)  =  logit(p')  +  ln(w)  =  logit(p') - 2.9957

Divide the second line by the first and f_1/f_0 cancels, leaving only the ratio of the two prior-odds terms:

true prior odds     pi  / (1 - pi )  =  0.01    / 0.99     =  0.010101
sampled prior odds  pi' / (1 - pi')  =  0.16807 / 0.83193  =  0.202020

ratio  =  0.010101 / 0.202020  =  0.05  =  w      exactly

so  o = w · o',  and taking logs:  ln(o) = ln(o') + ln(w)
                                   ln(0.05) = -2.9957

Downsampling negatives is an intercept bug and nothing else. An intercept is the constant term of a linear model, so the whole distortion is one number added to every score rather than anything shape-changing.

Three consequences follow immediately, and the third is the expensive one:

• AUC is bit-for-bit unchanged, because adding a constant to every logit cannot reorder anything.
• The ranking is untouched, for the same reason.
• Every probability is nonetheless wrong by a factor of 20 in the odds (1/w = 1/0.05 = 20).

The table below converts that constant odds error into probability error across the score range. Read a row left to right: the model emits p', you convert to odds, multiply by w = 0.05, convert back. The right column is what the auction should have been given.

Walk the last row by hand, because it is the most alarming one:

model emits            p' = 0.950
odds                   0.950 / 0.050        =  19.000
multiply by w = 0.05   19.000 x 0.05        =   0.950
back to probability    0.950 / (1 + 0.950)  =   0.48718

A downsampled model that says 0.95 means 0.49. Feed that into eCPM = 1000 · p · bid and you have inflated the ad’s value by 0.95 / 0.487 = 1.95x — which, by Three places the bias does not cancel, is money handed from the publisher to that advertiser on every mixed auction.

Note that the odds error is exactly 20x on every row, but the probability error is not: it is 19.05x at p' = 0.050 and only 1.95x at p' = 0.950. Offline metrics returns to why that distinction matters when you say this out loud.

5.2 What downsampling costs, statistically

The obvious objection: surely throwing away 94% of your data costs accuracy. It costs about 9%, for a reason worth understanding.

Fisher information measures how much a dataset actually tells you about a parameter. The more sharply the likelihood peaks around the true value, the tighter your estimate, and standard errors shrink as the square root of it.

For a logistic model the information is sum_i p_i(1 - p_i) x_i x_i^T. Ignore the x x^T part — it is the same for both cases here. The factor that matters is p(1 - p), and it is the whole argument: a row the model is already confident about, with p near 0, contributes almost nothing. Seeing another certain non-click teaches you nothing you did not already know.

p(1 - p) is largest at p = 0.5 and collapses toward the ends. So moving the base rate from 1% toward 17% makes every individual row far more informative, even though there are far fewer of them. The block below is that trade, in two numbers:

undownsampled:   p ≈ 0.01     ->  p(1-p) = 0.00990   per row, over  1.00 N rows
downsampled:     p ≈ 0.168    ->  p(1-p) = 0.13985   per row, over  0.0595 N rows

information retained  =  0.13985 x 0.0595 / 0.00990  =  0.840
standard errors inflate by  1/sqrt(0.840)            =  1.091

Reconstruct that line by line:

information per row, before   0.01  x 0.99  =  0.00990
information per row, after    0.168 x 0.832 =  0.13985     14.1x more per row
rows kept                                   =  0.0595      16.8x fewer rows

total information ratio  =  14.1 x 0.0595  =  0.840        so 84 % survives
standard error ratio     =  1 / sqrt(0.840) =  1.091        so 9 % wider

You keep 84% of the Fisher information on 5.95% of the rows, at a 9% cost in standard errors.

That is the whole argument for negative downsampling, and it is arithmetic rather than folklore. Near p = 0 each row carries very little curvature; concentrating the sample near p = 0.17 buys 14x more information per row, which nearly pays for the 16.8x fewer rows.

5.3 The three ways teams get this wrong

A short bug catalogue, ordered so that the last one is the one that survives code review.

1. Forgetting the offset. This one is invisible rather than loud. AUC is unchanged, because the offset cannot reorder anything. Log loss computed on the downsampled validation set also looks fine, because that set has the same wrong base rate the model was fitted to. Nothing fails until revenue does. Fix: validate log loss and COPC on an un-downsampled holdout, always.

2. Double-correcting. Applying ln(w) and then fitting an isotonic map on downsampled validation data corrects the same bias twice — the isotonic fit sees data at the 16.8% base rate and pushes the already-corrected scores back up. Fix: apply the offset first, calibrate on true-prior data second. Order matters.

3. A segment-varying w with a global offset. This is the interesting one, because it survives code review.

You will want to vary w: keep every negative on rare inventory where data is scarce, and downsample hard on the head where you have billions of rows. That is a sensible instinct. But the moment w varies by segment and the offset does not, each segment is shifted by the wrong constant — and you have manufactured a segment-structured calibration error by hand. That is precisely the failure The real cost heterogeneous error and the optimizers curse prices at billions.

Fix: keep w global, or carry it per row and apply ln(w_i) per row.

The code below implements the correction three ways — as a probability transform, as a logit offset, and as the COPC check that catches you having skipped it — then recomputes the The correction derived table row by row. The asserts at the bottom are the section’s claims: exactly 20x on the odds everywhere, but 19.05x on price at the bottom of the range and only 1.95x at the top.

from math import log

def downsample_correct(p_sampled: float, w: float) -> float:
    """Undo negative downsampling at rate w (positives kept at rate 1).

    Exact whenever p(x | y) is unchanged by the sampling, which is true for
    uniform random negative sampling. NOT true if negatives were sampled
    non-uniformly (e.g. only impressions from the top slot).
    """
    odds = p_sampled / (1.0 - p_sampled) * w
    return odds / (1.0 + odds)


def logit_offset(w: float) -> float:
    """The same correction as a constant added to every logit: ln(w)."""
    return log(w)


def copc(clicks: int, predicted: list[float]) -> float:
    """Clicks Over Predicted Clicks. 1.0 is calibrated. Compute it PER SEGMENT:
    a global 1.00 routinely hides a 2x spread across competing segments."""
    s = sum(predicted)
    return clicks / s if s else float("nan")


# --- the section 5.1 table, recomputed rather than transcribed ---------------
W = 0.05
print("  p'      odds o'   o = w.o'    true p    p'/p on PRICE")
for p_sampled, expected in ((0.050, 0.00262), (0.100, 0.00552), (0.168, 0.01000),
                            (0.300, 0.02098), (0.500, 0.04762), (0.800, 0.16667),
                            (0.950, 0.48718)):
    o_sampled = p_sampled / (1.0 - p_sampled)
    p_true = downsample_correct(p_sampled, W)
    print("  %.3f   %7.4f   %8.5f   %8.5f      %5.2fx"
          % (p_sampled, o_sampled, o_sampled * W, p_true, p_sampled / p_true))
    assert abs(p_true - expected) < 5e-6                 # the table's last column
    # exactly 20x on the ODDS, at every single row -- which is the claim
    assert abs((p_sampled / (1 - p_sampled)) / (p_true / (1 - p_true)) - 20.0) < 1e-9

print("logit offset ln(w) = %.5f" % logit_offset(W))
assert abs(logit_offset(W) + 2.99573) < 1e-5

# the price error is 19.05x at the bottom of the range and 1.95x at the top, so
# "20x wrong on every price" is false in the direction that matters: the
# expensive impressions are the ones where it is least wrong.
assert abs(0.050 / downsample_correct(0.050, W) - 19.05) < 0.01
assert abs(0.950 / downsample_correct(0.950, W) - 1.95) < 0.01

5.4 The calibration map, fit per segment

The layer that stands between the ranker and the money is the concrete answer to the whole of Why calibration is non negotiable here. One constant fixes the bias you introduced on purpose; a fitted map per segment fixes the bias you did not.

Why one constant is not enough

The ln(w) offset of The correction derived fixes the one bias that is provably constant in logit space. It does nothing for the segment-structured error of The real cost heterogeneous error and the optimizers curse — the vertical that runs hot, the daypart that runs cold — because that error is not a single number.

What the COPC dispersion of The production instrument copc per segment points at instead is a per-segment monotone recalibration. The recipe, in four constraints:

• Fix a segment grid in advance (vertical × device × slot × hour bucket).
• Fit one one-dimensional map from predicted probability to observed rate per cell.
• Fit it on that cell’s own un-downsampled holdout.
• With the ln(w) offset already applied — calibrating on downsampled data is the The three ways teams get this wrong double-correction.

Isotonic regression, and the algorithm that fits it

Isotonic regression is that map. “Isotonic” means order-preserving, so the fitted function may bend as much as it likes but may never go downhill.

The algorithm is pool-adjacent-violators (PAVA). It works like this:

1. Sort the held-out predictions in ascending order.
2. Sweep along them looking for two neighbouring groups whose observed rates go the wrong way round — a lower-predicted group that actually clicked more.
3. Merge that pair into one block and use their combined average.
4. Repeat until nothing is out of order.

What comes back is the non-decreasing step function closest to the observed rates. Non-decreasing is the entire point: it repairs calibration without ever reversing two candidates inside a segment. COPC moves to 1 and nothing gets reordered.

What isotonic actually costs: it manufactures ties

Be precise here, because the sentence people finish that paragraph with — “so grouped AUC is untouched” — is measurably false. And this is the only positive claim the chapter makes for isotonic; the two alternatives below are rejected on their own grounds rather than in its favour.

PAVA works by merging blocks, so the map it returns is non-decreasing but not strictly increasing. It sends whole intervals of raw score to a single step value. Different raw scores come out equal. It manufactures ties the raw score did not have.

Run the fit in the code below on the section’s own other cell — 60,000 held-out rows from the generator in the code — and measure it:

That AUC move is small. But this chapter treats ±0.004 of AUC as decision-relevant (Offline metrics), so “untouched” is the wrong word. The honest claim is that isotonic never reverses a pair, at the cost of tying some.

The magnitude is distribution-dependent — a smoother score distribution and a larger holdout both collapse less — so quote a tie rate you measured on your own holdout rather than this one, and say what generated it.

The operational consequence is one line of serving code: break ties with the pre-calibration score, not arbitrarily. An auction whose top is tied after calibration otherwise picks its winner by whatever order the candidate list happened to arrive in. That is a silent, unlogged source of exactly the The real cost heterogeneous error and the optimizers curse selection error the calibration layer exists to remove. The raw score is the natural tiebreak, because it is the finest ordering you have and isotonic is by construction consistent with it.

The two cheaper alternatives, and why each fails

• Temperature scaling divides every logit by one learned constant. One parameter cannot carry corrections that differ per segment, and per-segment is the whole problem.
• Platt scaling fits a logistic curve to the scores. That imposes an S-shape which the segment biases have no reason to follow.

The code

The block below builds five synthetic segments, each with a different systematic logit bias — exactly the situation a single global offset cannot touch — then fits one isotonic map per segment and measures what happened. Two things to watch: the COPC spread collapsing from about 2x to about 1.04x, and then the adversarial half, which measures the ties that collapse buys.

# Continues the section 5.3 block, which is where `copc` is defined. It is
# restated in one line below so that a reader who copies only THIS block still
# gets output: a block that runs only because an earlier one is still in scope
# is a block the reader cannot check.
import math, random, bisect, statistics

def copc(clicks, predicted):                       # section 5.3, restated to stand alone
    s = sum(predicted)
    return clicks / s if s else float("nan")

def isotonic_fit(pred, label):
    """Pool-adjacent-violators: the non-decreasing map from predicted
    probability to observed rate. Returns (edges, values) -- a step function
    where values[j] applies to every prediction <= edges[j]. Fit PER SEGMENT,
    each on that cell's own un-downsampled, offset-corrected holdout."""
    order = sorted(range(len(pred)), key=lambda i: pred[i])
    blocks = [[float(label[i]), 1, pred[i]] for i in order]   # [sum_y, count, x_right]
    out = []
    for b in blocks:
        out.append(b)
        while len(out) > 1 and out[-2][0] / out[-2][1] >= out[-1][0] / out[-1][1]:
            sy, c, xr = out.pop()          # violation: merge the two blocks
            out[-1][0] += sy; out[-1][1] += c; out[-1][2] = xr
    return [b[2] for b in out], [b[0] / b[1] for b in out]

def isotonic_apply(model, p):
    edges, values = model
    return values[min(bisect.bisect_left(edges, p), len(values) - 1)]

def _sig(z): return 1.0 / (1.0 + math.exp(-z))
def _logit(p): return math.log(p / (1.0 - p))

# five cells of the section 3.4 grid, each with a SYSTEMATIC logit bias a global
# offset cannot touch: mobile.gaming under-predicts, desktop.retail over-predicts.
seg_bias = {"mobile.gaming": -0.34, "desktop.retail": 0.42,
            "mobile.retail": -0.10, "desktop.news": 0.22, "other": 0.47}

def segment_holdout(beta, n, seed, base=0.05):
    r = random.Random(seed)
    tr_p, tr_y, te_p, te_y = [], [], [], []
    for k in range(n):
        z = _logit(base) + r.gauss(0, 0.9)      # heterogeneous true CTR in-cell
        y = 1 if r.random() < _sig(z) else 0
        p_pred = _sig(z + beta)                  # the model's systematically biased score
        (tr_p if k % 2 else te_p).append(p_pred)
        (tr_y if k % 2 else te_y).append(y)
    return tr_p, tr_y, te_p, te_y

pre, post = [], []
for s, beta in enumerate(seg_bias.values()):
    trp, trl, tep, tel = segment_holdout(beta, 120_000, seed=s)
    cmap = isotonic_fit(trp, trl)                # fit on the holdout half
    pre.append(copc(sum(tel), tep))              # COPC before  (copc: section 5.3)
    post.append(copc(sum(tel), [isotonic_apply(cmap, p) for p in tep]))   # after

print("pre-calibration  COPC per cell  " + " ".join("%.2f" % c for c in pre))
print("post-calibration COPC per cell  " + " ".join("%.3f" % c for c in post))
print("spread %.2fx pre -> %.2fx post   dispersion %.1fx smaller"
      % (max(pre) / min(pre), max(post) / min(post),
         statistics.pstdev(pre) / statistics.pstdev(post)))

assert max(pre) / min(pre) > 1.8                 # the ~2x section-3.4 spread, pre-fix
assert all(0.97 <= c <= 1.03 for c in post)      # every cell now in band
assert statistics.pstdev(post) < statistics.pstdev(pre) / 8   # dispersion ~17x smaller

# --- what the map costs: PAVA MERGES blocks, so it manufactures ties --------
# `tep`, `tel` and `cmap` are the last cell of the loop above. The adversarial
# case, not the confirming one: the claim under test is "grouped AUC is
# untouched", so measure the ties and the grouped AUC rather than the COPC that
# was going to move anyway.
cal = [isotonic_apply(cmap, p) for p in tep]
rng = random.Random(11)
pairs = [(rng.randrange(len(tep)), rng.randrange(len(tep))) for _ in range(200_000)]
strict = [(i, j) for i, j in pairs if tep[i] != tep[j]]     # ordered BEFORE calibration
tied = sum(1 for i, j in strict if cal[i] == cal[j])        # ...and tied after it
auctions = [[rng.randrange(len(cal)) for _ in range(100)] for _ in range(1_500)]
tied_top = sum(1 for a in auctions
               if [cal[i] for i in a].count(max(cal[i] for i in a)) > 1)

def grouped_auc(score, label, groups):
    """AUC inside each auction, averaged. Ties count 0.5, which is the whole
    point: a tie is neither a correct nor an incorrect ordering."""
    tot, n = 0.0, 0
    for g in groups:
        pos = [score[i] for i in g if label[i] == 1]
        neg = [score[i] for i in g if label[i] == 0]
        if not pos or not neg:
            continue
        tot += sum(1.0 if p > q else 0.5 if p == q else 0.0
                   for p in pos for q in neg) / (len(pos) * len(neg))
        n += 1
    return tot / n

auc_raw = grouped_auc(tep, tel, auctions)
auc_cal = grouped_auc(cal, tel, auctions)

print("distinct raw predictions        %8d" % len(set(tep)))
print("distinct calibrated predictions %8d" % len(set(cal)))
print("strictly ordered pairs -> tied   %7.1f%%" % (100.0 * tied / len(strict)))
print("100-candidate auctions, tied top %7.0f%%" % (100.0 * tied_top / len(auctions)))
print("grouped AUC  %.5f -> %.5f  (%+.4f)" % (auc_raw, auc_cal, auc_cal - auc_raw))

assert len(set(cal)) < len(set(tep)) / 100   # a step function, not a relabelling
assert tied > 0                              # ties the raw score did not have
assert tied_top > 0.10 * len(auctions)       # and they reach the auction winner
assert auc_cal < auc_raw                     # so grouped AUC is NOT untouched
assert abs(auc_cal - auc_raw) < 0.004        # ...but it moves less than a launch decision

The five cells reproduce the The production instrument copc per segment spread: pre-calibration COPC runs 1.35 / 0.70 / 1.09 / 0.83 / 0.66, a 2x range. After one isotonic map per cell, every segment lands inside [0.98, 1.03] and the dispersion collapses about 17x.

Why that matters in dollars: dispersion is the direct s_e estimate of The real cost heterogeneous error and the optimizers curse. Collapsing it 17x is the move from the s_e = 0.25 row of that table to below the s_e = 0.10 row — the row pair worth $2.42B a year. That is the per-segment isotonic node in the serving diagram (Serving and the model size ceiling derived from the auction budget), and it is why the calibration layer, not the network, is where the auction’s money is made.

Three operational details decide whether this works in production rather than in a notebook.

1. The grid must be coarse enough that every cell fills. A cell with a few hundred positives a day fits a map to noise and calibrates nothing. Below a traffic floor, fall back to the parent cell — vertical first, then the global map.
2. The map must be refit on the same cadence as the model (Training online learning and the freshnessstability tradeoff). A stale calibration map on a fresh model is itself exactly the segment-structured error it was built to remove.
3. The auction must carry the pre-calibration score alongside the calibrated one and break ties on it, because of the block collapse measured above. The per-segment isotonic node emits a step function, and a tied top is otherwise resolved by candidate list order.

Assumptions in this section, and the load-bearing one. The downsampling rate w = 0.05 is a free choice and nothing depends on its value, only on its being recorded and applied. The load-bearing assumption is the one the code’s docstring states and it is easy to violate: negatives must be dropped uniformly at random. The correction is exact because uniform sampling leaves the feature distribution within each class untouched and changes only the mixing proportion. Sample negatives non-uniformly — keep only top-slot impressions, say, or keep all negatives on rare inventory as The three ways teams get this wrong warns — and the distortion is no longer a constant in log-odds, so no single offset can undo it. Secondary: the base rate of 1% sets the size of the offset and nothing else.

6. Features: high-cardinality categoricals at billions of values

What the model eats is dominated by one problem: cardinality — the number of distinct values a field can take. Several of these fields have billions, which makes the naive answer of one learned vector per value cost more memory than the serving machine has.

An embedding is a short vector of learned numbers standing in for a categorical value, so that a model which can only do arithmetic on numbers can work with things like “advertiser 4,182,113.” Give each distinct value its own row of 16 numbers and the model can learn what that value means.

The table below lists the fields and how many distinct values each has. Read the second column as “how many rows the embedding table would need.” It is a memory problem before it is a modeling problem.

Now price it. Each value gets 16 numbers, and each number is fp32 — 32-bit floating point, four bytes. So one row costs 16 × 4 = 64 bytes, and a field costs distinct values × 64 bytes:

ad / creative      2.0e8 x 16 x 4  =   12.8 GB
user id            2.0e9 x 16 x 4  =  128.0 GB     <- dominates
campaign            5.0e7 x 16 x 4 =    3.2 GB
publisher           2.0e6 x 16 x 4 =    0.13 GB
hashed crosses      2^26 x 16 x 4  =    4.3 GB
                                       --------
                                       148.4 GB

One field is 86% of the bill — 128.0 / 148.4 = 0.863. That is where the next subsection starts.

6.1 Do not embed the user id

One decision removes 86% of the memory bill and improves the prediction at the same time, which is rare enough to be worth deriving rather than asserting.

Start with the number that makes a user-id embedding sound reasonable: the mean user has 900 impressions and 9 clicks over 90 days. Nine clicks is enough to learn a 16-dimensional vector from.

That mean is a lie, because impression counts are Zipfian — a few users account for an enormous share of the volume and the long tail accounts for almost none, so the median user sits far below the mean. Look at percentiles instead. p50 is the median user; p90 is the user with more activity than nine out of ten others:

p50 user, 90 days:   ~40 impressions,   0 clicks
p90 user, 90 days:   ~2,400 impressions, 18 clicks

The median user has zero clicks. Not few — zero.

That is fatal, and here is the mechanism. Gradient descent only updates an embedding when a row containing that value appears in a batch, and it only pushes the vector upward when the label is 1. With no positive labels, the user’s vector receives no positive gradient at all. What is left pulling on it is the regularizer — the penalty term that pulls unconstrained parameters toward zero. So the vector simply decays to the default and predicts the prior.

For the median user, a 16-dimensional identifier embedding converges to the regularizer’s prior and carries no information. That is the same underdetermination argument as Why collaborative filtering structurally fails here, reached from a different direction.

Replace it with user history features — aggregates instead of an identity:

• Click counts and recency by category, advertiser and vertical over 1, 7 and 30 days.
• A pooled embedding of the last 50 ads the user interacted with.
• The user’s own CTR shrunk toward a device-and-geography prior, meaning blended with that prior in proportion to how little evidence the user has.

That removes 128 GB — 86% of the embedding footprint — and improves the median user’s prediction, because a feature that generalizes beats an id that does not. A user with 40 impressions and no clicks still has a device, a geography and a category history; all of those transfer from other users. Their id transfers from nobody.

6.2 Hashing, and why the collisions land where they do not matter

The remaining identifiers are handled by giving up on having one row per value — and the resulting mistakes land almost entirely where nobody notices.

Hashing a feature means running its value through a hash function and using the result as an index into a fixed-size table. An unbounded set of identifiers maps into a bounded array, and there is no dictionary to build, store or keep in sync.

The price is collisions: two different identifiers landing in the same row and therefore sharing one embedding. Hash the remaining ids into 2^24 = 16.8M buckets (ml/01).

The naive objection is “5e7 ads into 1.7e7 buckets means about three ads per bucket, so every embedding is a blend of three unrelated things.” That is true and irrelevant, because impressions are Zipfian. Almost all of those three-way collisions involve ids nobody ever sees. What matters is collisions among the head — the ads that actually get shown.

head ads (top 1 %, carrying ~60 % of impressions)     5.0e5
buckets                                               1.68e7

P(a given head ad shares a bucket with another head ad)
   =  1 - exp(-5.0e5 / 1.68e7)  =  0.0294

That formula is the standard birthday-collision approximation. Each of the other 5.0e5 − 1 head ads independently misses your bucket with probability 1 − 1/B, so the chance all of them miss is (1 − 1/B)^H, which for large B is very close to exp(−H/B). One minus that is the chance at least one hits you:

H / B          =  5.0e5 / 1.68e7   =  0.0298     expected head neighbours
exp(-0.0298)   =  0.9706                          chance of no head neighbour
1 - 0.9706     =  0.0294                          chance of at least one

2.9% of head ads collide with another head ad. The rest of the collisions are tail-on-tail, and a tail id’s embedding was going to be noise anyway.

The damage is also asymmetric in a way that helps. A bucket’s learned weight is dominated by whichever colliding feature has more occurrences. So a head-tail collision leaves the head correct and gives the tail the head’s prior — which is a better estimate than the tail had on its own.

The fix for the 2.9%

A frequency-based hybrid table: dedicated rows above an impression threshold, hashed shared rows below it. The head gets exclusive rows and the tail shares:

ids above 10,000 lifetime impressions      1.2e6   ->  dedicated rows
everything else                            2.0e8   ->  hashed into 2^24
head collision rate                        0.0294  ->  0
memory  1.2e6 x 16 x 4  +  2^24 x 16 x 4  =  0.077 + 1.07  =  1.15 GB

Compare that 1.15 GB against the 148.4 GB the naive table cost. Two decisions got you there: dropping the user id (128 GB) and hashing everything else (the rest).

Two operational details keep it correct over time:

• Re-evaluate the impression threshold as identifiers churn, which they do at 40% a month. A threshold set once goes stale with the inventory.
• Reset a bucket’s embedding whenever its set of identifiers changes, rather than letting it be inherited. Otherwise a retired ad’s learned weight silently becomes a new ad’s starting prior — a bug that looks like a mysteriously well-performing new creative and gets celebrated instead of fixed.

6.3 One row, populated

Every section above has assumed a populated feature row; none has shown one. So here is one ad, its feature values filled in, scored end to end — the price that came out and where it ranked. Everything it consumes is already in the chapter — the The correction derived correction table, the w = 0.05 sampling rate, and the three competing bids of Three places the bias does not cancel.

The heavy ranker turns those 45 sparse fields and 60 dense values into one number, and that number is a logit in the downsampled world — the 16.8%-base-rate world of Extreme imbalance negative downsampling and the correction it forces, not the real one.

The code below takes it from there in four steps, and each step is a section of this chapter:

1. Sum four contributions — one intercept, two wide crosses, one deep term — into a raw logit.
2. Turn the logit into p_sampled with the sigmoid.
3. Apply the The correction derived ln(w) correction to get the real p.
4. Run the The ml objective auction: multiply by the bid, compare against three rivals and a floor, and compute the GSP price.

Then it does the adversarial version: the same candidate with step 3 skipped.

import math

# --- one candidate, scored end to end ---------------------------------------
contributions = {                       # the raw logit, IN THE DOWNSAMPLED WORLD
    "intercept (downsampled base rate)":            -1.85,
    "wide: advertiser 4182113 x domain 91177":      +0.62,
    "wide: vertical(mobile.gaming) x hour(21)":     +0.18,
    "deep: MLP over the 780-wide input":            +0.20,
}
z_sampled = sum(contributions.values())
p_sampled = 1.0 / (1.0 + math.exp(-z_sampled))

W = 0.05                                # section 5: negatives kept at 5%
odds_true = p_sampled / (1.0 - p_sampled) * W          # o = w * o'
p_true = odds_true / (1.0 + odds_true)

BID = 1.20
ecpm = 1000.0 * p_true * BID            # section 2: eCPM for a CPC advertiser

# the section 3.2 field, unchanged: two rival eCPMs and the publisher floor
rivals = {"ad A (CPC, true CTR 1.80%)": 21.60,
          "ad B (CPM, bid known exactly)": 19.00,
          "ad C (CPC, true CTR 1.20%)": 15.60}
FLOOR = 8.00
board = sorted(list(rivals.items()) + [("THIS candidate", ecpm)],
               key=lambda kv: -kv[1])
rank = [name for name, _ in board].index("THIS candidate") + 1
price_per_click = board[1][1] / (1000.0 * p_true)      # GSP, section 2

for name, v in contributions.items():
    print("  %-44s %+.2f" % (name, v))
print("raw logit (downsampled)      %+.4f" % z_sampled)
print("p_sampled                     %.5f   <- what the network emits" % p_sampled)
print("p_true after ln(w) offset     %.5f   <- what the auction may multiply" % p_true)
print("eCPM = 1000 x p x bid        $%.2f" % ecpm)
for name, v in board:
    print("   %-32s $%6.2f" % (name, v))
print("rank %d of %d, floor $%.2f, GSP price per click $%.4f"
      % (rank, len(board), FLOOR, price_per_click))

assert abs(p_sampled - 0.29943) < 1e-5     # ~0.300, the section 5.1 table's fifth row
assert abs(p_true - 0.02092) < 1e-5        # which that row corrects to 0.02098
assert p_sampled / p_true > 14             # 14.3x on PRICE; exactly 20x on the odds
assert abs(p_sampled / (1 - p_sampled) * W - p_true / (1 - p_true)) < 1e-12
assert abs(ecpm - 25.11) < 0.01
assert rank == 1 and ecpm > FLOOR          # it wins, and it clears the floor
assert price_per_click < BID               # GSP: the winner pays less than it bid

# and the adversarial case, which is the whole of section 5: skip the offset
ecpm_uncorrected = 1000.0 * p_sampled * BID
print("eCPM if ln(w) is forgotten   $%.2f   (%.1fx)"
      % (ecpm_uncorrected, ecpm_uncorrected / ecpm))
assert ecpm_uncorrected > 350.0            # $359 against a $19 CPM rival

Read the output in order.

The network emits 0.29943. That is a probability in a world where 95% of the negatives were thrown away, so it is not a probability of anything real.

The ln(w) offset turns it into 0.02092. That is the The correction derived table’s 0.300 -> 0.02098 row, reached from the logit side rather than the odds side.

Multiplied by the $1.20 bid, that is an eCPM of $25.11. It beats ad A’s $21.60, clears the $8.00 floor, and wins the slot at a generalized-second-price cost of $1.03 per click against a $1.20 bid — the same conversion worked in What the auction does with p.

Now forget the offset. The same candidate posts an eCPM of $359.32, 14.3x too high. Do not stop at “too high” — follow it through the Three places the bias does not cancel mechanism:

• It now beats every CPM bid it meets, including ones genuinely worth more than it.
• The publisher who was promised $359.32 per thousand impressions realizes $25.11.
• The advertiser is not overbilled. GSP divides by the same inflated p, so their price per click falls. They are delighted.

That last point is why nobody complains and the leak runs quietly for months.

That single row is the whole chapter: the same model, the same ranking, the same AUC, and a fourteen-fold error in the only number anybody prices against.

Assumptions in this section, and the load-bearing one. The load-bearing assumption is that impressions are Zipfian — that a small head carries most of the volume. Both results in the section depend on it and neither survives without it: it is why a hashed table works at all (collisions concentrate in a tail whose embeddings were noise anyway), and it is why the user-id embedding is worthless (the median user has no clicks). If traffic were uniform across ads and users, the memory argument and the collision argument would both reverse. Secondary: 16 dimensions per embedding, and the 10,000-impression threshold for a dedicated row, both of which trade memory against accuracy smoothly and neither of which changes a conclusion.

7. Models: LR, FM, deep, and what each one fixes

The ranking model is built in four steps, under one discipline: each step must fix a named limitation of the one before it rather than simply being newer. LR is logistic regression, the linear model that turns a weighted sum of features into a probability; FM is a factorization machine, introduced in Factorization machines what they actually fix; MLP is a multi-layer perceptron, the ordinary stack of dense layers with non-linearities that “deep” usually means here.

7.1 Logistic regression on hand-built crosses, and its exact limit

The baseline comes with a precise shape of what it cannot do, and that shape is what everything after it is buying.

A cross is a feature made by pairing two others — “advertiser 4182 and domain 91177” treated as a single value with its own weight. It exists because a linear model has no other way to say that a combination behaves differently from its parts.

Logistic regression is linear in the features it is given. It can learn “this advertiser is good” and “this domain is good,” but there is no term anywhere in w · x that says “this advertiser is good specifically here.” You have to build that cross by hand.

Build it and count how many of the possible pairs you will ever see:

advertiser x domain          5.0e6 x 1.0e7  =  5.0e13 possible pairs
training rows, 90 days                      =  1.8e12
distinct pairs actually observed            ~  4.0e11
coverage                                    =  0.8 % of possible pairs

The coverage line is 4.0e11 / 5.0e13 = 0.008, or 0.8%. Note that even with 1.8 trillion training rows you cannot do better: there are 28 times more possible pairs than you have rows, so most pairs are unobservable in principle, not just in practice.

A cross weight is learnable only for pairs you have observed. For every other pair the feature contributes exactly zero. That is not a bug — it is what a cross is.

LR with crosses is therefore a memorization device: excellent on the head, mute on everything it has not seen, and its parameter count grows with the product of cardinalities.

7.2 Factorization machines: what they actually fix

Factorization machines repair exactly the limitation above — though the reason they usually get credit is not the reason they actually work.

A factorization machine keeps logistic regression’s structure but changes how pairs are represented. Instead of one free weight w_ij per pair of features, it gives every feature a short learned vector v_i and computes the pair’s weight as the inner product <v_i, v_j> — multiply the two vectors element by element and add up the results.

k below is the length of those vectors (16 here) and n the number of features:

y  =  w_0  +  sum_i w_i x_i  +  sum_{i<j} <v_i, v_j> x_i x_j

Three terms, left to right. w_0 is the intercept, the base rate before any feature speaks. sum_i w_i x_i is ordinary logistic regression — one weight per feature. sum_{i<j} <v_i, v_j> x_i x_j is the new part: every pair of active features contributes, and its contribution is decided by how aligned their two vectors are.

The parameter count changes dramatically:

parameters:   n · k    instead of   n^2 / 2
n = 2.05e9 features, k = 16    ->   3.3e10   vs   2.1e18

Substitute: 2.05e9 × 16 = 3.3e10 for the FM, against (2.05e9)^2 / 2 = 2.1e18 for one free weight per pair. That is a factor of 64 million.

The parameter count is the advertised win. The real win is statistical.

v_i receives gradient from every row containing feature i, not only from the rare rows containing the specific pair (i, j). So a pair with zero co-occurrences still gets a non-trivial prediction — the model estimates it from n_i + n_j observations instead of from 0.

That is generalization to unseen crosses, and it is exactly the thing Logistic regression on hand built crosses and its exact limit showed LR structurally cannot do. It fixes the 99.2% of pairs LR is mute on.

FMs are also cheap to score, because the pairwise sum has a closed form that avoids ever enumerating the pairs:

sum_{i<j} <v_i,v_j> x_i x_j  =  0.5 · sum_f [ (sum_i v_if x_i)^2 - sum_i v_if^2 x_i^2 ]

45 active features, k = 16:
   naive   45 x 44 / 2 x 16  =  15,840 mults
   closed  2 x 45 x 16       =   1,440 mults      11x

The naive count is “every pair, times k dimensions”: 45 × 44 / 2 = 990 pairs, times 16, is 15,840 multiplications. The closed form only ever sweeps the 45 features twice per dimension: 2 × 45 × 16 = 1,440. That is 11x fewer, and it grows linearly in the feature count rather than quadratically.

What a factorization machine still cannot do sets up the next step, and there are two limits:

• It is degree-2: it models pairs of features and nothing higher. Three-way interactions are invisible to it.
• Each feature carries one vector no matter which kind of feature it is meeting. Yet “user × hour” and “user × publisher” plausibly need different aspects of the same user, and one vector has to serve both.

Field-aware factorization machines (FFM) fix the second limit by giving each feature a separate vector per field it can interact with, where a field is a whole column such as “publisher” or “hour.” The cost is F times the memory for F fields, and roughly 4x the scoring time — which the table in Deep and why the wide part does not go away shows as 38 microseconds per candidate against FM’s 9.

7.3 Deep, and why the wide part does not go away

What settles the architecture is that its two halves are kept for opposite reasons, not because more is better.

A multi-layer perceptron over concatenated embeddings learns interactions of any order. But it does so through a smooth function approximator, and smoothness is precisely the wrong property for memorizing that one specific advertiser-domain pair converts at 8x the base rate — smoothness is exactly what spreads that fact over its neighbours instead of keeping it where it belongs.

Deep models generalize and blur. Linear crosses memorize and do not generalize. The wide-and-deep architecture exists because those are different jobs, not because two models are better than one.

The two metrics in the comparison table

Neither is standard outside this domain, so define both before reading the table.

RIG is relative information gain: the fraction by which the model beats a constant prediction of the base rate.

RIG  =  1 - logloss / H

H  =  entropy of the label  =  -(0.01·ln 0.01 + 0.99·ln 0.99)  =  0.056002 nats

H is the average uncertainty of a 1% coin, measured in nats — the natural-logarithm unit of information, the same role bits play for base-2 logs. It is the log loss you would get by always predicting exactly 1%. So RIG answers “what fraction of the label’s uncertainty did the model actually remove?” Substituting the best row of the table:

DCN v2 log loss  =  0.05131
RIG  =  1 - 0.05131 / 0.056002  =  1 - 0.9162  =  0.0838  =  8.38 %

ECE is expected calibration error: bucket the predictions by predicted value, compare each bucket’s mean prediction against its observed click rate, and average the gaps weighted by bucket size. It is the direct measurement of the property Why calibration is non negotiable here says is worth billions.

Lower is better for log loss and ECE; higher is better for RIG. us/candidate is microseconds of scoring time per candidate.

Read the table with the ECE column first, not the log-loss column.

DCN v2 in the last row is Deep and Cross Network, version 2: a design that builds feature crosses up to a chosen degree inside the network, so you get the memorizing half without hand-writing the 40 crosses.

Now compare “Deep only” against “FM, k = 16”, which is the row pair that decides the architecture.

              log loss    RIG      ECE
FM, k = 16     0.05221   6.77 %   0.0026
Deep only      0.05204   7.07 %   0.0071

deep wins log loss by 0.00017, and RIG by 0.30 points
deep loses ECE by  0.0071 / 0.0026  =  2.7x

Deep-only wins the number everyone reports and is 2.7x worse calibrated. The mechanism is known: deep nets trained to convergence on cross-entropy are systematically overconfident (Why modern deep nets are overconfident the mechanism).

By the The real cost heterogeneous error and the optimizers curse dollar table, an ECE regression of that size costs more revenue than a 0.3-point RIG gain earns. In this problem, a model that wins log loss and loses calibration is a losing model, and log loss alone will not tell you.

One last intuition. RIG going from 1.02% to 8.38% sounds enormous, and log loss going from 0.05543 to 0.05131 sounds like noise. They are the same fact, stated twice.

The reason is the base rate. At 1% the entropy is 0.056 nats, so every log loss on this problem is pinned near 0.056 and the fourth decimal place is the whole game. A 0.004 move in log loss is a 7-point move in RIG. That is exactly why you report RIG and not raw log loss — the raw number hides the signal behind two leading digits that never change.

Assumptions in this section, and the load-bearing one. The comparison table is a measurement on this system rather than an assumption, so what is assumed is narrower: that the ECE column and the RIG column can be traded against each other using the dollar table of The real cost heterogeneous error and the optimizers curse. That exchange rate is the load-bearing assumption of the architecture choice, because it is the entire reason a model that wins log loss is rejected. It holds only if the ECE difference reflects segment-structured error rather than a uniform shift — a uniform shift would cancel in the auction, by The naive argument and why it fails. So the honest gate is not ECE but per-segment COPC dispersion, which is what Offline metrics makes the launch criterion.

8. Training: online learning, and the freshness/stability tradeoff

How often should the model be updated? This is one of the few places in this repo where the answer is “continuously” — a claim that has to be earned by measuring what staleness costs, naming the optimizer that makes continuous training practical, and spending equal time on what continuous training takes away: a rollback point.

The distribution moves hourly: campaigns launch and exhaust budgets, creatives rotate, news events shift traffic mix, and the time of day changes who is on the platform. A model retrained once a day is stale by the time it deploys.

8.1 FTRL-Proximal, and the two things it buys

The optimizer earns its place with two specific properties, neither of which is “it converges faster.”

FTRL-Proximal stands for Follow The Regularized Leader, proximal variant. It is an online optimizer: at each step it keeps the weights that would have been best on everything seen so far plus a penalty term, and it updates them one example at a time rather than in epochs over a fixed dataset.

Thing one: per-coordinate learning rates

Below, g_{s,i} is the gradient for feature i at step s, and eta_{t,i} is that feature’s own learning rate at step t. alpha and beta are tuning constants.

per-coordinate learning rate:   eta_{t,i}  =  alpha / (beta + sqrt( sum_{s<=t} g_{s,i}^2 ))

Read the denominator: it accumulates the squared gradients that feature has ever received. The more a feature has been updated, the bigger that sum, and the smaller its learning rate becomes. Each feature anneals on its own schedule.

That matters because feature frequencies span nine orders of magnitude, so a single global learning rate cannot exist. Set it for the head and the tail never moves; set it for the tail and the head oscillates.

head feature:  1e10 occurrences   ->  large accumulated g^2  ->  tiny steps (converged)
tail feature:  30 occurrences     ->  small accumulated g^2  ->  large steps (learning)

ratio of effective rates  ≈  sqrt(1e10 / 30)  =  1.8e4

The sqrt in that ratio comes straight from the sqrt in the denominator: accumulated squared gradient grows roughly in proportion to occurrence count, and the learning rate is one over its square root. So an 18,000x spread in effective learning rate falls out of a 300-million-x spread in frequency, automatically, with no tuning.

Thing two: exact zeros

L1 regularization penalizes the sum of the absolute values of the weights. Unlike its squared cousin L2, which only shrinks weights toward zero, L1 drives them all the way to zero.

FTRL’s closed-form update applies L1 in a way that produces true sparsity. A weight that never earns its place is never materialized — no row is allocated for it in the serving process at all, as opposed to being stored as a very small number that still costs four bytes.

buckets touched at least once, 90 days     4.1e9
nonzero after FTRL L1 (lambda_1 tuned)     3.2e8      ->  12.8x
serving memory at 4 bytes/weight           1.3 GB

The last line is 3.2e8 weights × 4 bytes = 1.28e9 bytes = 1.3 GB. Without the L1 sparsity it would be 4.1e9 × 4 = 16.4 GB, which does not fit alongside everything else a serving process holds.

Without L1 sparsity the linear component does not fit in a serving process, so this is a deployment constraint expressed as a regularizer. Note the ordering of that argument. It is not “L1 improves generalization,” which would be marginal here at 1.8e12 training rows.

8.2 What freshness is worth, measured

“The model should be fresh” is an instinct until it carries a dollar figure, and the experiment that produces one has to be permanent to work.

Run a permanent staleness holdback: a small slice of traffic served by a deliberately frozen model, refreshed weekly.

It has to be permanent and it has to be online. Offline log loss on a fixed test set understates staleness, because the test set is frozen too — it cannot show you the traffic the stale model has never seen, which is precisely where the damage is.

RPM below is revenue per mille, revenue per thousand impressions, the headline online metric of Online metrics and the ab. Read the table down the last column: that is the price of every hour you do not retrain.

Notice the RPM column falls much faster than the log-loss column reads — another instance of the Deep and why the wide part does not go away point that at a 1% base rate the fourth decimal of log loss is the whole signal.

A 24-hour-old model costs 5.1% of revenue: 0.051 × $32.9B = $1.7B a year on this traffic. That number, not a preference for streaming architectures, is what justifies the operational cost of online training.

8.3 The stability side, and the incident it prevents

Freshness comes with a bill, best presented as a specific outage worked through minute by minute — ending with the one reordering that shortens it most.

Online learning has no epoch — no pass over a fixed dataset that ends at a known point — and therefore no natural place to roll back to. A corrupted hour is simply absorbed into the weights.

The incident below is, at bottom, a train/serve skew incident (Trainserve skew and point in time correctness). slot_position did not go missing everywhere at once — it went missing on the serving side, in one region, while the training pipeline carried on computing it exactly as before. The model was being updated against a feature whose serve-time value no longer matched its train-time definition, which is the definition of skew, and the whole 31% COPC excursion is the model absorbing that mismatch. This is worth labelling because it is what makes the Trainserve skew and point in time correctness feature-distribution alarm the first detector in the list rather than a nicety: a serve-side field that has silently become constant is visible in a feature histogram within one minute and in COPC after forty.

09:14  upstream logging change drops the `slot_position` field in region EU
09:14  feature assembly defaults the missing field to 0  ( = "slot 1" )
09:20  online updates begin attributing slot-1 examination to ad relevance
09:54  EU COPC = 1.31    (predictions inflated 31 %)
10:02  automatic guardrail fires on segment-COPC dispersion
10:03  rollback to the 09:00 checkpoint; EU traffic pinned to the batch model
10:40  fix deployed; online training resumes from the checkpoint
       measured revenue impact:
         EU is 22 % of $90 M/day          ->  $1.18 M of revenue in those 86 min
         COPC 1.31  =>  ln(1.31) = 0.27 of segment-structured log-odds error
         §3.3 at s_e = 0.27               ->  10 % of RPM  ->  $121 k

The last four lines of that timeline are the damage estimate, and they are worth reconstructing:

EU share of a $90M/day platform     0.22 x $90M          =  $19.8M/day
the incident ran 09:14 -> 10:40     86 min / 1,440 min   =  0.0597 of a day
revenue exposed                     $19.8M x 0.0597      =  $1.18M

COPC of 1.31 in log-odds            ln(1.31)             =  0.27
look 0.27 up in the section 3.3 table (between 0.25 -> 9.0 %
and 0.40 -> 18.8 %)                                      ->  ~10 % of RPM lost
damage                              0.10 x $1.18M        =  ~$121k

Four controls, in order of value:

1. Checkpoint every 15 minutes and keep 48 (that is 12 hours of history). Without a saved snapshot there is nothing to roll back to.
2. Keep a batch-trained model live as a fallback, so there is somewhere to send traffic during the 40 minutes you are debugging.
3. Guard the input batch. Reject any update whose positive rate, missing-feature rate or number of distinct feature values deviates more than 5 standard deviations (5 sd) from the trailing hour.
4. Clip per-coordinate updates, so no single bad batch can move any single weight far even if it slips through the guard above.

The guardrail that matters is on the input batch, not on the output metric, because the output metric is 40 minutes late and the input check is free. That reordering is what turns the incident above from a 5-hour outage into an 86-minute one.

Assumptions in this section, and the load-bearing one. The freshness table is measured, not assumed, but the architecture it justifies rests on one thing: that the incident above is recoverable, which requires the batch fallback model to be permanently maintained. That is the load-bearing assumption, and it is a cost people forget to count — the batch pipeline has to be kept correct and warm even though it never serves in the steady state, which is why Alternatives considered and rejected rejects “pure online learning, no batch model” outright. Secondary: the 15-minute checkpoint interval and the five-standard-deviation input threshold, both of which trade false alarms against exposure and neither of which changes the shape of the design.

9. Offline metrics

What has to be true offline before a model is allowed near live traffic? A conjunction rather than a single number — because the most popular single number in ranking is structurally blind to this chapter’s main failure.

Two more abbreviations: PR-AUC is the area under the precision-recall curve, and a reliability diagram is the plot behind ECE — predicted probability on one axis, observed rate on the other, with a perfectly calibrated model lying on the diagonal.

Two of those rows deserve to be raised before anyone asks about them, because both concern AUC and both are ways a rising AUC can mean nothing. The first is about which comparison AUC is scoring, and the second is about what AUC cannot see at all.

1. Global pooled AUC measures the wrong comparison. Pooled means every impression from every request is thrown into one ranking; grouped, or per-auction, means AUC is computed inside each request and averaged. Pooled, the model is rewarded for telling a gaming ad on a mobile game at 9pm apart from a B2B ad on a news site at 9am — a distinction the auction never makes, because it only ever compares candidates within one request, where context is held fixed.

global pooled AUC           0.812
per-auction (grouped) AUC   0.594

The 0.812 is mostly measuring context, and the auction holds context constant. A model can gain 0.004 of pooled AUC and lose grouped AUC; only the second is revenue.

2. AUC cannot see the failure that costs the most.

AUC is invariant to every strictly increasing transform of the score. Three of this chapter’s expensive bugs are exactly such transforms, pooled or grouped:

• The missing The correction derived ln(w) offset.
• The The real cost heterogeneous error and the optimizers curse segment biases.
• The Delayed conversions and the bias they inject delayed-label offset.

A perfect-AUC model with a missing ln(w) offset is wrong by a factor of 20 on the odds of every impression.

State it as an odds ratio, and be careful here, because “20x wrong on every price” is the version people say and it is false. 20x is exact on p/(1-p). It is only approximate on p itself, and the approximation gets worse the higher the score. The The correction derived table has the whole range:

The price error shrinks toward the top of the score range, which is where the expensive impressions are. A candidate the model is confident about is the one it misprices least, so the dollar damage is not the flat 20x the slogan implies.

It is still enormous, though, and here is why: the mass of the score distribution sits near the p' = 0.168 downsampled prior, so the typical impression is mispriced 16.8x. And the fix is a single added constant.

The interview move is: say the odds number, then say where on the range the price number lands.

The launch gate is therefore a conjunction: RIG must improve and per-segment COPC dispersion must not regress. Either condition alone ships losers — RIG alone ships the miscalibrated model, dispersion alone ships the model that learned nothing.

Assumptions in this section, and the load-bearing one. The load-bearing assumption is that per-segment COPC dispersion measured offline predicts the s_e that the auction experiences online. Everything in the launch gate depends on it, and it can fail in one specific way worth naming: the segment grid is fixed in advance, so a calibration error that is structured along a dimension not in the grid — a new publisher, an unmodelled device class — is invisible to the dispersion metric and fully visible to the auction. That is the argument for revisiting the grid when traffic composition changes, not just when the model does. Secondary: the pooled and grouped AUC values, which illustrate the gap rather than carry any decision.

10. Online metrics and the A/B

Once the model is live, the question becomes what to measure and how to measure it honestly. The interesting difficulty is not statistical power — the platform has far too many users for that to bind — but the fact that advertisers’ budgets couple the two arms of the experiment together.

Sizing the experiment is easy, and that is exactly the trap.

MDE is the minimum detectable effect: the smallest true difference the experiment is designed to catch. sd is standard deviation and sigma^2 its square. The 16 sigma^2 / delta^2 formula is the standard two-sample sizing rule at 80% power and 5% significance.

baseline RPM                      $4.50
per-user daily revenue sd         $9.20     (heavy right tail)
MDE 0.5 % relative                $0.0225

n per arm  ≈  16 sigma^2 / delta^2  =  16 x 84.64 / 0.0225^2  =  2.68 M users

Substituting each piece:

MDE       0.5 % of $4.50           =  $0.0225      <- delta
sigma^2   9.20^2                   =  84.64
16 x 84.64                         =  1,354.24
0.0225^2                           =  0.00050625
1,354.24 / 0.00050625              =  2,675,000 users per arm

as a share of the platform  2.68e6 / 500e6  =  0.54 %

2.68M users is about half a percent of a 500M-DAU platform, so the experiment is powered within a day.

Which means the danger here is not power, it is peeking — checking a result repeatedly and stopping as soon as it looks significant. A metric that reads out hourly invites twenty looks, and twenty looks take the type-I error rate (the chance of declaring a win when nothing is really there) from 5% to well past 20% (Multiple testing). Fix the horizon in advance, or use a sequential test that budgets its error across the looks it will take.

The interference problem

Interference means the two arms of the experiment are not independent, so measuring one tells you partly about the other. It is specific to ads and it is severe.

Randomizing users does not isolate advertisers here, because budget is shared across arms. Follow the mechanism:

1. Treatment ranks better, so it wins more auctions for advertiser X.
2. Advertiser X’s daily budget therefore drains faster.
3. When the budget runs out, X stops being eligible — in both arms.
4. Control loses access to the same ads, and looks worse.

So the measured effect partly reflects a resource transfer between arms rather than a real improvement. And it does not replicate at 100% rollout, because there is no control arm left to take budget from.

A switchback design randomizes time rather than users: the whole marketplace runs treatment for an interval, then control, alternating. That is the only way to measure a change that alters the marketplace itself, such as moving the reserve price, because there is no way to give one user a different floor than another.

Volunteer that a 1% RPM lift on a 0.5% user-randomized arm will not fully replicate at 100%, and that the mechanism is budget interference rather than novelty.

Assumptions in this section, and the load-bearing one. The sample-size calculation assumes per-user daily revenue with a standard deviation of $9.20 and a heavy right tail, and that assumption barely matters because the platform has two orders of magnitude more users than the test needs. The load-bearing assumption is instead the one behind every number the experiment produces: that a budget-split or switchback design actually removes the interference. It removes the budget channel; it does not remove marketplace effects such as the treatment changing what the competing bids look like, which is why the last row of the table exists and why a marketplace-wide change gets a switchback rather than a user split.

11. Serving, and the model-size ceiling derived from the auction budget

Follow one ad request from arrival to response, and then keep the promise the chapter has been making since its first paragraph: derive the maximum number of parameters the ranking model may have, from a deadline set by an external exchange, before anyone has said a word about quality.

The diagram below is the whole serving path. Read it top to bottom: a request arrives, gets narrowed from every campaign on the platform down to 1,000 candidates, then to 100, then to one winner. Two things stand out on a first pass. The funnel narrows before it gets expensive — the cheap model runs on 1,000 candidates and the expensive one on 100. And the calibration layer sits between the ranker and the auction, which is the structural claim of this entire chapter.

flowchart TD
    REQ(["Ad request<br/>user · page · slot"]) --> TGT["Targeting retrieval<br/>inverted index over<br/>geo · demo · keyword · audience"]
    TGT --> ELIG["Eligibility<br/>budget remaining · pacing<br/>frequency cap · brand safety"]
    ELIG --> C1["~1,000 candidates"]
    C1 --> LIGHT["Light ranker<br/>sparse LR · 30 kFLOP each<br/>30 MFLOP total"]
    LIGHT --> C2["top 100"]
    C2 --> EMB[("Embedding store<br/>1.15 GB hybrid table<br/>4,500 random reads")]
    EMB --> HEAVY["Heavy ranker<br/>Wide & Deep · 1.45 M params<br/>2.9 MFLOP x 100 = 291 MFLOP<br/>batched into ONE GEMM"]
    HEAVY --> CORR["Calibration layer<br/>+ ln(w) downsample offset<br/>+ per-segment isotonic"]
    CORR --> AUC{"Auction<br/>eCPM = 1000 · p · bid<br/>vs CPM bids vs floor"}
    AUC --> EXPL{"1.5 % exploration<br/>randomize among top 20"}
    EXPL --> SERVE(["1-5 ads + prices"])
    SERVE --> LOG[("Impression log<br/>+ slot · + explore flag<br/>+ the p that was served")]
    LOG -.->|windowed join with clicks| ONL["Online trainer<br/>FTRL · 15-min checkpoints"]
    ONL -.->|weights| HEAVY

    style LOG fill:#1d3557,color:#fff
    style EMB fill:#2d6a4f,color:#fff
    style CORR fill:#bc6c25,color:#fff
    style ONL fill:#9d0208,color:#fff

The colours are not decoration and they are not this chapter’s own invention. They are the key published in The ladder, republished here so this chapter reads on its own, with the box each one lands on in this design:

Two boxes deliberately lost their colour, and the reasons are worth stating. The auction is uncoloured because it owns nothing and answers no read; it is arithmetic over numbers other boxes produced, and colouring the most important box on the page is a habit worth breaking. And the exploration node is uncoloured because it had danger-red and does not deserve it: red means the rung you cannot undo, and 1.5% exploration is a config change you can turn off between requests. It is also a thing this chapter recommends, at a priced 0.73% of revenue (The feedback loop you only observe clicks on ads you showed), so painting it as the danger was backwards.

Now trace the diagram box by box, because every later subsection is an argument about one of them.

Ad request · user · page · slot. The entry node: one request, identifying who is browsing, which page they are on, and which slot is for sale.

Targeting retrieval. Looks up which campaigns are even eligible, using an inverted index — a lookup keyed by attribute value that returns the ads matching it — over geography, demographics, keywords and audience lists.

Eligibility. Removes anything that cannot legally or commercially be shown right now: campaigns out of budget, campaigns the pacer is holding back, ads this user has already seen too often (the frequency cap), and ads barred from this page by brand safety rules. About 1,000 candidates survive.

Light ranker. Cuts 1,000 down to the top 100, at 30 thousand floating-point operations each — 30 MFLOP in total, or 2.5% of the request’s 1.2 GFLOP budget.

Embedding store. The 1.15 GB hybrid table of Hashing and why the collisions land where they do not matter. Roughly 4,500 individual rows are read at random per request, because each of the 100 surviving candidates needs its 45 sparse fields looked up: 100 × 45 = 4,500.

Heavy ranker. Scores all 100 as a single matrix multiplication, 2.91 MFLOP each, 291 MFLOP in total.

Calibration layer. Turns those raw scores into prices — the ln(w) offset of The correction derived followed by the per-segment isotonic map of The calibration map fit per segment.

Auction. Compares the calibrated eCPMs against CPM bids and the floor.

Exploration. On 1.5% of requests this node fires. Its rule is randomize among top 20: the auction’s winner is set aside and one of the twenty leading candidates is shown instead. The feedback loop you only observe clicks on ads you showed prices what that costs and what it buys.

Response. 1-5 ads + prices goes back to the exchange.

Impression log. Everything served is written down, with the slot, the exploration flag, and crucially the probability that was actually served. Without that last field, no later analysis can reconstruct what the model believed.

Online trainer. A windowed join with clicks attaches each impression’s outcome and feeds it to FTRL.

Two labels in that diagram deserve a note because they are the whole point of the section. GEMM is a general matrix-matrix multiply — the single dense linear-algebra operation the entire heavy ranker collapses into once the 100 candidates are batched, and Two implementation notes that matter more than they sound shows what happens if you do not batch them. And the arrow from the log back to the trainer is dotted because it is asynchronous: nothing on the request path waits for it.

11.1 The budget, decomposed

All 100 milliseconds have to be accounted for, so that the 20 that remain for scoring are a residual rather than a preference. RTT is round-trip time, the network delay out and back.

Read the block as a subtraction, not a plan. The 100 ms at the top is imposed by the ad exchange; every line below it is spent by something that is not the model; whatever is left over is what the model gets.

exchange-imposed auction wall clock                  100 ms
  network RTT, bidder <-> exchange                    40 ms
  ------------------------------------------------------------
  bidder budget                                       60 ms
  targeting retrieval + eligibility filters           18 ms
  feature assembly + embedding gather                 14 ms
  auction, pacing, price computation, encode           8 ms
  ------------------------------------------------------------
  MODEL SCORING BUDGET                                20 ms

Check the arithmetic: 100 − 40 = 60, then 60 − 18 − 14 − 8 = 20. Nobody chose 20 ms. It is what nothing else wanted.

11.2 The ceiling, derived

Converting that 20 ms into a parameter count takes four lines, and the result is the single most useful number in the chapter for shutting down an unbuildable proposal.

Three facts go into the derivation, and each is worth stating separately because each is an assumption you can be challenged on.

1. One request is handled by one CPU core. Splitting a 20 ms unit of work across cores costs more in coordination than it saves.
2. A modern server core sustains roughly 60 GFLOP/s — 60 billion floating-point operations per second — on a batched 32-bit matrix multiply.
3. A dense network costs about 2 floating-point operations per parameter, per input: one multiply and one add. That is where the 2 x params line comes from.

Now multiply and divide:

FLOP budget      =  0.020 s x 60e9         =  1.2 GFLOP per request
heavy candidates                           =  100
FLOP per candidate                         =  12 MFLOP
dense MLP FLOPs  =  2 x params
                                              ------------
                      params ceiling       =  6 M

Step by step: 20 ms at 60 GFLOP/s is 0.020 × 60e9 = 1.2e9 operations. Split across 100 candidates that is 1.2e9 / 100 = 12e6 operations each. At 2 operations per parameter that is 12e6 / 2 = 6e6 parameters. Six million, and not one more.

What actually ships is well under that. The block below counts the parameters layer by layer — each layer’s cost is inputs × outputs, which is the size of its weight matrix:

input:  45 sparse fields x 16-dim  +  60 dense   =  780
780 -> 1024 -> 512 -> 256 -> 1

  780 x 1024  =   798,720
 1024 x  512  =   524,288
  512 x  256  =   131,072
  256 x    1  =       256
                ----------
                 1,454,336 params   =  2.91 MFLOP / candidate
100 candidates   =  291 MFLOP       =  4.85 ms      (24 % of the budget)

The input width is 45 × 16 = 720 from the sparse embeddings plus 60 dense values, so 780. The last three lines are 1,454,336 × 2 = 2.91e6 FLOP per candidate, × 100 = 291e6 FLOP for the batch, and 291e6 / 60e9 = 0.00485 s = 4.85 ms, which is 4.85 / 20 = 24% of the scoring budget.

And here is what does not fit, at the two sizes people most often propose:

100 M params  ->  200 MFLOP/candidate x 100  =  20 GFLOP  =  333 ms     17x over
  1 B params  ->                                            3,333 ms   167x over

100e6 params × 2 = 200e6 FLOP per candidate, × 100 candidates = 20e9, and 20e9 / 60e9 = 0.333 s. Against a 20 ms budget that is 333 / 20 = 17x over.

A 100M-parameter model is not “a bit slow” here, it is 17x over a hard external deadline. Model size is decided before quality is discussed — the same shape of argument as genai-system-design/02, where a 70B model spends the whole budget 2.8 times over on decode.

11.3 The asymmetry to state out loud

“The model is too big” and “the table is too big” are completely different sentences, and the reason why is the idea from this chapter most worth carrying into other systems.

Compare the two blocks of parameters this system holds. Ignore the totals for a second and look only at how much of each gets touched on a single request:

embedding table   1.15 GB (hybrid) — of which  4,500 rows x 64 B  =  288 KB is read
MLP weights       5.8 MB           — of which  ALL of it is traversed, once per batch

The embedding table is 200 times larger than the MLP, and each request reads 0.025% of it (288 KB / 1.15 GB). The MLP is tiny, and each batch reads 100% of it.

The lookup table may be enormous because it is indexed; the dense network must be tiny because it is traversed. Parameters you index are free; parameters you multiply are not.

That is why the answer to “make the model bigger” in ad serving is always “make the embedding table bigger,” and it is the single most transferable idea in this chapter.

11.4 Two implementation notes that matter more than they sound

Two engineering decisions stand between the 4.85 ms figure above and fiction.

1. Batch the 100 candidates into one matrix multiply.

Scored one at a time, the 5.8 MB of weights has to stream out of memory 100 times:

5.8 MB x 100 candidates       =  580 MB of memory traffic
580 MB at ~100 GB/s           =  5.8 ms

against 4.85 ms of arithmetic -> the memory traffic costs MORE than the math

That is the difference between being memory-bound, where the chip sits waiting on data, and compute-bound, where it waits on arithmetic. The 60 GFLOP/s figure in The ceiling derived assumes compute-bound. So batching is not an optimization — it is what makes the whole derivation true rather than optimistic.

2. Compute the user-side representation once, outside the per-candidate loop.

It is about 30% of the input width and it does not depend on which candidate is being scored. Computing it inside the loop does the same work 100 times.

Assumptions in this section, and the load-bearing one. The load-bearing assumption is 60 GFLOP/s sustained on one core, because the parameter ceiling is directly proportional to it. It is worth stating as an assumption rather than a fact, since it depends on the batching above being done properly and on the arithmetic being 32-bit. Notice, though, how little it matters to the conclusion: a core would have to be 17 times faster than assumed for a 100M-parameter model to fit, and that is not a measurement error, it is a different decade of hardware. Secondary: the 100-candidate cut, the 20 ms budget’s internal split, and the 100 GB/s memory bandwidth used in the batching argument.

12. Scale and cost

Pricing the fleet establishes which constraint is actually binding — and the answer is not money.

The block below goes from daily request volume to a server count in five steps: FLOPs per request, FLOPs per second at peak, FLOPs per server, servers needed, servers actually provisioned.

2.0e10 requests/day  ->  231,000 QPS average,  ~700,000 peak

per request:  light 1,000 x 30 kFLOP  =   30 MFLOP
              heavy   100 x 2.91 MFLOP =  291 MFLOP
                                          ---------
                                          321 MFLOP   ->  5.35 ms of one core

peak fleet  =  700,000 x 321 MFLOP  =  2.25e14 FLOP/s  =  225 TFLOP/s
per server  =  32 cores x 60 GFLOP/s  =  1.92 TFLOP/s
            ->  117 servers of pure scoring
            ->  ~350 with retrieval, feature assembly, and headroom
            +   ~40 for the sharded, replicated embedding tier

Annual cost, in four lines:

Now put that against $32.9B of revenue. The three lines below are the same fleet cost compared against three things it could buy:

serving fleet as a share of revenue   =  $1.37 M / $32.9 B  =  0.004 %
value of a 0.1 % revenue improvement  =  $32.9 M / yr    ->  24x the fleet
gap between s_e = 0.25 and s_e = 0.10 =  $2.42 B / yr    ->  1,800x the fleet

Read those as ratios: a 0.1% revenue improvement is worth 24 entire serving fleets (32.9M / 1.37M = 24), and closing the calibration gap of The real cost heterogeneous error and the optimizers curse is worth 1,800 of them (2,420M / 1.37M = 1,766).

Cost is not the binding constraint in ad ranking; latency is. The team will spend anything for a tenth of a RIG point and still cannot ship a 100M-parameter model, because the constraint is a 20 ms deadline set by someone else’s exchange.

That last line — calibration worth 1,800x the entire serving fleet — is why this chapter is about calibration. Note how differently it reads from Scale and cost, where the whole system cost $230k and the feature store was the expensive part.

13. Failure modes

The five ways this system breaks share a single shape: the click log is not a sample of the world. It contains only ads that were shown, only in the slots they happened to occupy, only with the outcomes that had arrived by the time you looked, and only against creatives that advertisers chose to optimize toward whatever you rewarded last quarter. Each failure below is one of those four distortions plus the incident it causes.

13.1 The feedback loop: you only observe clicks on ads you showed

The first distortion comes from the log containing only ads that won. The standard fix has a price, and it pays for itself in a way that has nothing to do with the ads it explores.

A new ad enters with a prior — the campaign or advertiser mean. A pseudocount is how that prior is made to fade: an ad is treated as if it already had m impressions’ worth of the advertiser average, so its own data outweighs the prior only once it has been shown roughly m times. If that prior lands below the winning threshold, the ad never gets an impression, never accumulates evidence, and stays at the prior forever.

impressions needed before an ad's own signal outweighs the advertiser prior
   (shrinkage with pseudocount m = 700):  ~700 impressions

ads that reach 700 impressions within 7 days of launch        61 %
ads that never reach 700                                      24 %   <- invisible forever

A quarter of new inventory is decided by a prior it never gets to update. This is the Why offline ranking metrics disagree with online ctr loop with money attached, and no amount of offline modelling fixes it — the missing thing is data that does not exist.

The fix is exploration, and the right way to argue for it is to price it rather than to appeal to principle. Allocate 1.5% of auctions to a randomized winner among the top 20 candidates:

auction winner   argmax of 100      realized worth  E[ exp(v_best) ]
explore winner   random of top 20   realized worth  E[ exp(v_pick) ]
value ratio, by the same simulation as §3.3          0.52

cost  =  1.5 % x (1 - 0.52)  =  0.73 % of revenue  =  $240 M / yr

The 0.52 comes from the same Monte-Carlo as The real cost heterogeneous error and the optimizers curse: draw 100 candidate values, take the best, and compare its worth against a candidate picked uniformly from the top 20. A random pick from the top 20 realizes about half what the winner does. So:

share of auctions randomized          1.5 %
value given up on each of them        1 - 0.52  =  0.48
cost as a share of revenue            0.015 x 0.48  =  0.0073  =  0.73 %
in dollars                            0.0073 x $32.9B  =  $240M / yr

$240M a year is a real number and it has to be earned back. It is, three ways:

1. New ads ramp faster — they reach 700 impressions in 1.2 days instead of 9.
2. New-advertiser 30-day retention rises 4 points, which raises second prices through competition. More bidders means every winner pays more.
3. The randomized slice is the only unbiased dataset in the system.

The third is the one people undersell. Its propensity — the probability each ad had of being shown — is known exactly, because you chose it, rather than being an artifact of what the ranker happened to believe.

That makes it the ground truth for the position-bias estimation of Position bias, and for every off-policy evaluation you will ever run — meaning any attempt to estimate how a system you have not deployed would have performed, using logs from the system you did deploy.

Thompson sampling does the same job more cheaply. It draws each ad’s click rate from a Beta distribution fitted to that ad’s own successes and failures, then shows whichever ad drew highest. An ad with little data has a wide distribution and so draws high often; an ad with lots of data has a narrow one. So it explores in proportion to genuine uncertainty rather than uniformly.

Keep a small uniform slice regardless. Thompson sampling’s propensities are unpleasant to compute after the fact; the uniform slice’s are exact.

13.2 Position bias

The next distortion: the log records clicks without recording whether the user ever looked. There are two fixes, and they are not interchangeable.

Start with the measurement. Run the same ads in different slots and their observed click rate falls off a cliff. Divide each row by the top row and you get an estimate of how often users even looked at that slot:

slot   observed CTR   implied P(examined)
  1       4.2 %            1.00
  2       2.1 %            0.50
  3       1.3 %            0.31
  4       0.9 %            0.21

The third column is slot CTR / slot-1 CTR: 2.1 / 4.2 = 0.50, 1.3 / 4.2 = 0.31, 0.9 / 4.2 = 0.21. That assumes the ads themselves are equally good across slots, which is why the estimate has to come from a randomization experiment rather than from ordinary traffic.

A logged click is examined AND relevant, and the log only records the product. Train on raw clicks and the model learns that ads which happened to be placed in slot 1 are good — and the ranker is what placed them there. The loop is closed and self-confirming.

Here is the damage, in the chapter’s own currency:

naive model, COPC by slot:      1.31 / 0.88 / 0.71 / 0.64

A spread of 2x in COPC across slots is precisely the segment-structured calibration error The real cost heterogeneous error and the optimizers curse prices at billions, and it is entirely manufactured by the training data.

Two fixes, and they are not equivalent:

(a) Position as a feature at train time, held constant at serve time. If examination and relevance enter additively in logit space, logit p = f(x) + g(slot), then including slot lets the model put the examination effect in g and serving with slot = 1 for every candidate removes it. Cheap, and it works — but only to the extent the additive assumption holds, and it does not fully, because the ranker’s slot assignment is correlated with x.

(b) Inverse propensity weighting (IPW). Weight each impression by 1 / P(examine | slot). An impression in a slot users look at only a fifth of the time then counts for five ordinary impressions. It is the same inverse-of-the-sampling-rate idea as an importance weight, applied to examination instead of to sampling.

Estimate the propensities from a randomization experiment: swap slots 1 and 2 on 0.5% of traffic, or reuse the The feedback loop you only observe clicks on ads you showed exploration slice.

IPW is unbiased, and it costs variance. The block below prices that cost. The last line is the number to watch:

P(examine | slot)   1.00 / 0.50 / 0.31 / 0.21
weights  w = 1/P    1.00 / 2.00 / 3.23 / 4.76   on a slot mix of  0.40 / 0.30 / 0.20 / 0.10

E[w]    =  .40(1.00) + .30(2.00) + .20(3.23) + .10(4.76)  =  2.122
E[w^2]  =  .40(1.00) + .30(4.00) + .20(10.43) + .10(22.66) =  5.952

ESS / n  =  E[w]^2 / E[w^2]  =  4.503 / 5.952  =  0.756

Unpack it. The weights are one over the examination probabilities: 1/1.00 = 1.00, 1/0.50 = 2.00, 1/0.31 = 3.23, 1/0.21 = 4.76. The slot mix 0.40 / 0.30 / 0.20 / 0.10 is what share of impressions land in each slot; it sums to 1.

E[w] is the average weight and E[w^2] the average squared weight, both taken over that mix:

E[w]    =  0.40 x 1.00 + 0.30 x 2.00 + 0.20 x 3.23 + 0.10 x 4.76
        =  0.400 + 0.600 + 0.646 + 0.476   =  2.122
E[w^2]  =  0.40 x 1.00 + 0.30 x 4.00 + 0.20 x 10.43 + 0.10 x 22.66
        =  0.400 + 1.200 + 2.086 + 2.266   =  5.952
E[w]^2  =  2.122^2                          =  4.503
ESS / n =  4.503 / 5.952                    =  0.756

ESS is effective sample size: how many equally-weighted rows your unequally-weighted rows are actually worth. A handful of rows carrying most of the weight tell you less than their count suggests. If every weight were identical, E[w]^2 would equal E[w^2] and ESS/n would be 1.

You give up a quarter of your effective sample size in exchange for an unbiased estimate. At 1.8e12 rows that is a trade you take without hesitating. At 1e6 rows it is not — which is why IPW is standard in advertising and rare in small recommender systems.

Note what drives the cost: the spread of the weights, not their mean. An examination probability of 0.05 in a fifth slot would give it a weight of 20 and push ESS/n below 0.4 on its own. That is why the deepest slots are usually clipped — their weights capped at some maximum — rather than weighted honestly.

Apply both fixes together and the slot-level miscalibration is essentially gone:

with (a) + (b), COPC by slot:   1.02 / 0.99 / 0.97 / 0.95

Compare against the naive 1.31 / 0.88 / 0.71 / 0.64 above: a 2.05x spread becomes a 1.07x one.

13.3 Delayed conversions, and the bias they inject

The third distortion is that the log is incomplete at the moment you read it — the most expensive of the four, because the incompleteness differs by advertiser vertical — which means it becomes a calibration spread between bidders who compete with each other. pCVR is the predicted conversion rate, the probability that a click leads to a purchase or signup.

Clicks arrive in seconds. Conversions do not, and the attribution window truncates the label:

share of eventual conversions that have arrived:
   within 1 h     52 %
   within 24 h    79 %
   within 7 d     94 %
   within 30 d    99 %

Train on a 24-hour window and 21% of true positives are labelled negative — they converted, but not yet.

This is exactly the The correction derived prior shift, with a different cause. There, you deleted 95% of the negatives on purpose and the odds moved by w = 0.05. Here, the world deletes 21% of your positives by not having produced them yet, and the odds move by F(24h) = 0.79. Same algebra, same one-constant fix:

share of conversions arrived by 24 h    F(24h)  =  0.79
logit offset                            ln(0.79)  =  -0.236
so pCVR is under-predicted by roughly   21 %

That would be harmless if it were uniform, because The naive argument and why it fails showed uniform bias cancels. It is not uniform. The table below breaks F(24h) out by vertical — the fourth column is ln of the second:

The reason is the product, not the model: someone installs a mobile game in ten minutes and finances a car over three weeks.

That is a 62-point spread in calibration (-4% to -66%) between verticals that compete in the same auctions.

By Three places the bias does not cancel, travel and auto advertisers systematically lose auctions they should win, to gaming advertisers whose labels merely happen to arrive fast — and the platform loses the difference. This is the single most expensive quiet bug in conversion modelling.

The principled fix: model the delay instead of truncating it

Split the model into two heads:

• A conversion head predicting p, the probability the click converts eventually — with no deadline.
• A delay head predicting the distribution f of how long that takes, with cumulative distribution function F. F is the running total of f, so F(t) is the probability the conversion has arrived by time t. (F(24h) = 0.79 is the number in the table above.)

Then write the likelihood for a not-yet-converted impression honestly. A pending impression is not a negative — it is “either it never converts, or it converts later, and I cannot tell which yet”:

converted at delay d_i:            p_i · f(d_i | x_i)
not converted after elapsed t_i:   1 - p_i · F(t_i | x_i)

L  =  prod_{converted} p_i f(d_i)  ·  prod_{pending} [ 1 - p_i F(t_i) ]

Look at the second line, because it is the whole trick. Naive training would use 1 - p_i — “this did not convert.” The delayed-feedback likelihood uses 1 - p_i · F(t_i) instead: it only holds against the model the part of the conversion probability that should have arrived by now.

Substitute two elapsed times for an impression the model gives p = 0.30:

1 hour elapsed,  F(1h) small, say 0.02
   1 - 0.30 x 0.02  =  0.994   ->  costs almost nothing. Correct: we do not know yet.

30 days elapsed, F(30d) = 0.99
   1 - 0.30 x 0.99  =  0.703   ->  now it is real evidence against conversion.

The delayed-feedback model’s whole content is refusing to call a young impression a negative. The code block below asserts exactly that: a one-hour-old pending impression must cost near-zero loss, and a 30-day-old one must cost far more.

from math import exp, log

def delayed_feedback_nll(p, lam, converted, delay, elapsed):
    """Negative log likelihood for one impression under an exponential delay.

    p        P(converts eventually), from the conversion head
    lam      delay hazard rate, from the delay head (1/mean delay)
    converted  1 if the conversion has been observed, else 0
    delay      observed delay, valid only when converted
    elapsed    time since the click, valid only when not converted
    """
    eps = 1e-12
    if converted:
        return -(log(p + eps) + log(lam + eps) - lam * delay)
    survived = 1.0 - p * (1.0 - exp(-lam * elapsed))
    return -log(max(survived, eps))


def window_offset(share_converted_in_window):
    """Cheap per-vertical alternative: a prior-shift logit offset. Exactly the
    section 5.1 correction with w = F(window), applied per segment because the
    delay distribution is per segment."""
    return log(share_converted_in_window)


# --- the per-vertical table above, recomputed --------------------------------
print("vertical          F(24h)   pCVR bias   logit offset")
verticals = (("mobile games", 0.96, -0.041), ("retail", 0.81, -0.211),
             ("travel", 0.58, -0.545), ("auto / finance", 0.34, -1.079))
for name, f24, expected_offset in verticals:
    off = window_offset(f24)
    print("  %-14s  %.2f      %+4.0f %%      %+.3f" % (name, f24, 100 * (f24 - 1), off))
    assert abs(off - expected_offset) < 5e-4

assert abs(window_offset(0.79) + 0.236) < 5e-4        # the overall 21% truncation
spread = 100 * (verticals[0][1] - 1) - 100 * (verticals[-1][1] - 1)
print("spread between the fastest and slowest vertical: %.0f points" % spread)
assert abs(spread - 62) < 0.5                          # 62 points, and they bid together

# the adversarial case for the DELAYED-FEEDBACK likelihood: a young impression
# that has not converted must cost almost nothing, or the model calls it a
# negative and reinvents the truncation bias the offset was meant to remove.
young = delayed_feedback_nll(p=0.30, lam=1 / 48.0, converted=0, delay=0.0, elapsed=1.0)
old_ = delayed_feedback_nll(p=0.30, lam=1 / 48.0, converted=0, delay=0.0, elapsed=720.0)
print("pending impression NLL: 1 h elapsed %.5f   30 d elapsed %.5f" % (young, old_))
assert young < 0.01                                    # 1 h old: we genuinely do not know
assert old_ > 30 * young                               # 30 d old: now it IS evidence

The cheap version is the per-vertical logit offset in the table above. It is worth noting what it is: the downsampling correction of The correction derived, with w = F(window), applied per segment. Same derivation, different cause — which is why Extreme imbalance negative downsampling and the correction it forces is in this chapter and not a footnote.

13.4 Advertiser-side gaming, and why every fix moves the objective downstream

The last distortion is the only one caused by other people responding to your system rather than by a property of logging. pCTR is the predicted click-through rate, and a decile is a tenth of a ranked population, so “the top CTR decile” is the 10% of creatives with the highest click rates.

Optimize pCTR and you select for the creative that gets clicked, which is not the creative that is useful. Advertisers notice what you reward and build toward it.

The block below compares the highest-CTR 10% of creatives against the middle 10%. Every line says the same thing from a different angle: the ads that win on click rate are worse on everything that happens after the click.

creatives in the top CTR decile, versus the median decile:
   post-click conversion rate           -41 %
   back-button within 3 s               +3.1x
   ad "hide" rate                       +2.4x
   30-day advertiser renewal             -9 points

There are three standard fixes. What matters is that none of them is free — each swaps the gaming problem for a labelling problem, and the last column is the one to read:

Every defence against gaming moves the objective further down the funnel, and every step down the funnel makes the label sparser, later, and more segment-biased. Naming that tradeoff is the answer; picking a point on it is a business decision, and the usual one is pCTR · pCVR · value with a quality multiplier and a hide-rate guardrail.

Two adjacent problems belong to the same family and are worth raising in the same breath.

Invalid traffic (IVT) is automated activity — bots — that inflates click rates for specific publishers. The model then learns “this publisher is excellent” and sends them more inventory. So invalid traffic must be filtered upstream of training, not only upstream of billing, which is where teams usually stop.

The auto-bidder closes a loop with the advertiser’s own optimizer. It consumes pCVR to set bids, so a model change alters the bids, the bids alter which impressions are won, and those impressions become next week’s training data. You are not modelling a fixed world; you are modelling a world that reads your output.

Assumptions in this section, and the load-bearing one. The load-bearing assumption runs under Position bias and it is stated there as a caveat rather than as an assumption, so state it plainly: that examination and relevance combine additively in log-odds, logit p = f(x) + g(slot). Fix (a) is exactly and only valid if that holds, and the section notes it does not hold fully, because the ranker chooses slots based on the same features x that predict relevance. That is why fix (b) exists and why the two are applied together rather than as alternatives. Secondary assumptions: the exponential delay distribution in Delayed conversions and the bias they inject, which is a modeling convenience rather than a claim about how people buy things, and the pseudocount of 700 in The feedback loop you only observe clicks on ads you showed, which sets where the cold-start cliff sits but not that there is one.

13.5 Summary of the failure modes

Every failure above, plus five smaller ones, each with the mechanism that causes it, the instrument that would catch it, and the control that prevents it. Read the middle column as a monitoring checklist: a failure with no detector is one you will find out about from an advertiser.

14. Alternatives considered and rejected

Every design has a negative space: for each plausible alternative, the reason a competent person would propose it and the specific number that rules it out. NDCG in the first row is normalized discounted cumulative gain, the standard ranking quality metric, and SMOTE is synthetic minority over-sampling technique, the usual recipe for manufacturing extra positive rows.

15. Interviewer pushback

These are the ten hardest follow-up questions this design attracts, each with the answer said out loud and a note on what the question is really testing. Use them as a self-check: any answer that surprises you points at the section to re-read.

“Why does calibration matter? Ranking is all the auction needs.” Testing: whether you will assert the standard answer or check it. This is the question. The standard answer is actually wrong, so let me start there. A uniform multiplicative bias cancels exactly in a second price auction — every eCPM scales by k, the argmax is unchanged, and the price is eCPM_2 / (1000 · p_1), so the k cancels top and bottom. Calibration matters for three other reasons. First, the reserve price is an absolute number that the model is not in, so a 1.25x inflation shows ads below the floor — about 6% of auctions sit within 15% of the floor. Second, and this is the clean one, CPM advertisers do not go through the model at all, so every mixed auction compares a modeled quantity against a known one and there is nothing for the bias to cancel against: an 0.85x under-prediction on a $21.60 CPC ad hands the slot to a $19.00 CPM bid, which is a 12% loss on those impressions. Third, the auto-bidder and the budget pacer consume the probability directly, so a 1.25x bias makes the pacer throttle a campaign that was going to deliver fine.

“Fine, but you said uniform bias cancels. So how expensive is it really?” Testing: whether you can quantify the thing you just argued for. Very, because real error is never uniform — it is segment-structured, and an auction takes a maximum over 100 candidates rather than an average. That is the optimizer’s curse. Model log value as v ~ N(0, s_v^2) and the score as v + e with e ~ N(0, s_e^2); regression to the mean shrinks the winner’s true value by s_v^2/(s_v^2 + s_e^2), but revenue is E[exp(v)], not the exponential of an average, so I simulate the ratio rather than trust the sqrt(2 ln n) closed form — which overstates the max by a fifth at these sizes. With 100 candidates and a 1.8x spread in eCPM per standard deviation, a log-odds error of 0.25 costs 9.0% of revenue and 0.10 costs 1.7%. On $32.9B that gap is $2.42B a year. And none of it is visible to AUC, because AUC averages over all pairs while the auction consumes only the maximum — different statistics, and only one of them pays.

“Your CTR is 1%. How do you handle the imbalance?” Testing: whether imbalance triggers a reflex or a decision. Statistically I do not handle it at all — log loss is proper, the consumer is a probability, and there is no threshold to move, so there is nothing to fix. I downsample negatives at 5% purely for compute, which shrinks the training set 16.8x. And then I owe a correction, because downsampling negatives multiplies the posterior odds by exactly 1/w: logit(p) = logit(p') + ln(0.05), a constant shift of -2.996. Without it a model that says 0.95 actually means 0.49, which nearly doubles that ad’s eCPM. The reason downsampling is nearly free is the Fisher information: near p = 0.01 each row contributes p(1-p) = 0.0099 of curvature, and near 0.168 each row contributes 0.140. So I keep 84% of the information on 5.95% of the rows, at a 9% inflation in standard errors.

“What could go wrong with that correction?” Testing: whether you have shipped it. Three things, and only the third is interesting. Forgetting it — invisible, because AUC is unchanged and log loss computed on the downsampled validation set also looks fine, so I validate on an un-downsampled holdout. Double-correcting — applying ln(w) and then fitting isotonic on downsampled data. And the real one: letting the sampling rate vary by segment while the offset stays global. You will want to keep all negatives on rare inventory, and the moment w differs between segments and the offset does not, you have manufactured a segment-structured calibration error, which is exactly the failure the optimizer’s-curse math prices at billions. Either keep w global, or carry it per row and apply ln(w_i).

“Walk me from logistic regression to what you would actually ship.” Testing: whether each step fixes a named thing. LR is linear in what you give it, so “this advertiser works on this domain” requires an explicit cross — and a cross weight is only learnable for pairs you have observed, which is about 0.8% of the 5e13 advertiser-domain pairs. It is a memorization device, mute on everything unseen. FM replaces the free pairwise weight with <v_i, v_j>, which drops parameters from n^2/2 to n·k — but the real win is statistical: v_i gets gradient from every row containing feature i, not only rows containing the pair, so an unseen pair still gets a prediction. FM is degree-2 with one vector per feature regardless of the field it meets; FFM fixes that at F times the memory. A deep MLP gets arbitrary-order interactions but blurs, which is wrong for memorizing that one advertiser-domain pair at 8x base rate — so wide and deep, because memorization and generalization are different jobs. What I ship is wide and deep at 1.45M dense parameters, and I would look at the ECE column before the log-loss column: deep-only beats FM on log loss and is 2.7x worse calibrated, which by the auction math is a net loss.

“How big can the model be?” Testing: whether you derive constraints or accept them. The exchange gives 100 ms wall clock, network RTT takes 40, retrieval and eligibility take 18, feature assembly 14, auction and encoding 8 — so scoring gets 20 ms. One request runs on one core, and a core sustains about 60 GFLOP/s on batched GEMM, so the budget is 1.2 GFLOP. Divided by 100 heavy candidates that is 12 MFLOP each, and a dense MLP costs 2 × params, so the ceiling is 6M parameters. What I ship is 1.45M — 780 to 1024 to 512 to 256 — which is 2.9 MFLOP per candidate, 4.85 ms for the batch, a quarter of the budget. A 100M-parameter model is 333 ms, 17x over a deadline someone else sets. But note the asymmetry: the embedding table can be gigabytes because only 4,500 rows get read, while the MLP must be megabytes because every candidate traverses all of it. Parameters you index are free; parameters you multiply are not. So “make the model bigger” in ad serving always means “make the table bigger.”

“Your AUC went up 0.004. Ship it?” Testing: whether a good number gets interrogated. Not on that. Two problems. First, if that is pooled AUC it is mostly measuring whether the model can tell a mobile-game ad at 9pm from a B2B ad at 9am, and the auction never makes that comparison — it only ranks within one request, where context is fixed. My pooled AUC is 0.812 and my grouped AUC is 0.594, and only the second one is revenue. Second, AUC is invariant to every monotone transform of the score, which includes the downsampling offset, the per-segment biases, and the delayed-conversion truncation — the three things that actually cost money. So the gate is a conjunction: RIG has to improve and per-segment COPC dispersion must not regress. And I would report RIG rather than raw log loss, because at a 1% base rate the entropy is 0.056 nats, every log loss is near 0.056, and the fourth decimal is the entire signal.

“Your global COPC is 1.003. Are you calibrated?” Testing: whether you know what an average hides. No, and 1.003 is roughly what a badly calibrated model looks like, because the errors cancel by construction — the model was fit to minimize log loss over the pooled data, so the global mean prediction is nearly forced to be right. On my traffic the segments behind that 1.003 run from 0.66 to 1.34, a 2x spread across segments that compete in the same auctions. So I alert on per-segment COPC against a fixed grid of vertical, device, slot and hour bucket, and I treat the dispersion of segment COPC as the headline, because it is a direct estimate of the s_e in the optimizer’s-curse formula.

“Why online learning? Batch is simpler.” Testing: whether freshness is asserted or measured. Because I measured it with a staleness holdback — a small slice served by a deliberately frozen model. One hour old costs 0.3% of RPM, six hours 1.6%, twenty-four hours 5.1%, seven days 16.8%. At $32.9B, a day of staleness is $1.7B a year, which pays for a lot of streaming infrastructure. FTRL-Proximal for two specific reasons: per-coordinate learning rates, because feature frequencies span nine orders of magnitude and the effective rate ratio between a head feature and a 30-occurrence tail feature is about 18,000x, so a single global rate cannot exist; and L1 that produces exact zeros, which takes 4.1e9 touched buckets down to 3.2e8 and is what makes the model fit in a serving process. The cost is that there is no epoch boundary and therefore no rollback point, so I checkpoint every 15 minutes, keep a batch model live as a fallback, and — most importantly — put guardrails on the input batch rather than the output metric, because a 5-sigma shift in feature-null rate is free to detect at 09:15 and a COPC alarm arrives at 09:54.

“You are only shown ads you chose to show. How do you break the loop?” Testing: whether you will pay for exploration in an explicit currency. An ad needs about 700 impressions before its own signal outweighs the advertiser prior, and 24% of new ads never get there — they are decided forever by a prior they never update. I pay for it: 1.5% of auctions pick a random winner from the top 20. By the same optimizer’s-curse simulation, a random pick from the top 20 realizes about 52% of what the argmax of 100 would, so the cost is 1.5% × 48% = 0.73% of revenue, about $240M a year. It earns that back three ways — new ads ramp in 1.2 days instead of 9, new-advertiser retention rises 4 points which raises second prices through competition, and the randomized slice has known propensities by construction, so it is the ground truth for position-bias estimation and the only unbiased set I have for off-policy evaluation.

“Your pCVR is systematically low for travel advertisers. Why?” Testing: whether you connect delayed labels back to the auction. Attribution window truncation. 79% of conversions arrive within 24 hours overall, so a 24-hour label misses 21% of positives and shifts the logit by ln(0.79) = -0.236. That would be harmless if it were uniform, but it is not: 96% of mobile-game conversions land within 24 hours against 58% for travel and 34% for auto and finance, so the implied bias runs from -4% to -66%. That is a 62-point calibration spread between verticals that bid against each other, so travel loses auctions it should win to gaming advertisers whose labels merely arrive faster, and the platform eats the difference. The fix is to stop treating a young impression as a negative: model a conversion head and a delay hazard, and write the likelihood for a pending impression as 1 - p·F(t) rather than 1 - p. The cheap version is a per-vertical logit offset of ln F(window) — which is literally the downsampling correction with w = F(window), applied per segment.

16. Cheat sheet

Every load-bearing result in the chapter, compressed to one line each. If you can reconstruct the derivation behind each row, you have the chapter.

Next: 09 — Similar Listings — where the score is neither a rank nor a price but a retrieval key, “similar” turns out to mean substitutable, and the hard constraints are dates and availability rather than milliseconds.