How to design ad click-through prediction

The model’s output is a price, not a rank: each P(click) is multiplied by a bid, so calibration is a correctness requirement, not metric hygiene.

The one idea

• Ranking only cares about score order; an auction multiplies p by a bid and compares it to a fixed floor and a budget controller.
• Ranking is invariant to any monotone rescaling of p. Pricing and the floor check are not, and both cost money.
• Calibration: among all times the model says 0.02, ~2 in 100 really click. Separate from ranking.
• COPC = observed clicks / sum of predicted probabilities. 1.00 = calibrated, >1 under-predicts, <1 over-predicts.

The stack

• Light ranker: sparse LR, ~30 kFLOP, 1,000 candidates → top 100.
• Heavy ranker: wide-and-deep (780→1024→512→256→1), 1.45M dense params + 1.15 GB embeddings → one raw score in the downsampled world.
• Calibration layer (between ranker and money): ln(w) offset + per-segment isotonic map → true-base-rate P(click).
• Four bias-correctors: conversion+delay head, examination model (position), ad-level shrinkage prior, quality model.

Numbers to know

• Volume: 500M DAU × 40 impr = 2.0e10 requests/day; 231k QPS avg, ~700k peak.
• CTR ≈ 1%; CPC $0.45 → $90M/day, $32.9B/yr. Every “X% of revenue” = $32.9B × X%.
• Latency: ~20 ms scoring, inside 60 ms bidder, inside 100 ms exchange auction.
• Funnel: ~1,000 targeted → 100 heavy-scored → 1–5 shown.
• eCPM = 1000 · p · bid — the currency the auction compares CPC and CPM bids in.

Why calibration bites

• Naive “wrong ad wins” fails: a uniform multiplicative bias k cancels in GSP (price is a ratio, k on top and bottom cancels).
• It doesn’t cancel where a modeled number meets a non-modeled one: (a) the fixed reserve/floor, (b) CPM bids that skip the model, (c) the auto-bidder (bid = target_CPA · pCVR) and pacer, which use p as a number in its own right.
• Real error is segment-structured (per traffic cell), not uniform. The auction takes the max of 100 → optimizer’s curse: the winner’s error is disproportionately positive.
• Systematic error > random error: a hot segment wins auctions persistently, so losses accumulate one direction.

• Halving s_e from 0.25 → 0.10 recovers ~$2.42B/yr. Gate on per-segment COPC dispersion (a direct s_e estimate), never global COPC (errors cancel to ~1.003).

Objective and the two on-purpose biases

• Loss: pointwise log loss, a strictly proper scoring rule (unique minimizer = true probability). A pairwise ranking loss is disqualified: its minimizer is any monotone transform of truth.
• Negative downsampling (compute only): keep positives, keep negatives at w = 0.05. Correction is one constant: logit(p) = logit(p') + ln(w), with ln(0.05) = -2.9957. Odds off by exactly 1/w = 20×; AUC unchanged. Costs only ~9% wider standard errors (Fisher info: p(1-p) is ~14× larger at 17% than at 1%).
• Delayed conversions: a 24 h window mislabels 21% of positives as negative (ln F(24h) = ln 0.79). Same algebra, but non-uniform by vertical (−4% games to −66% auto/finance). Fix: conversion head + delay head, or per-vertical ln F(window) offset.
• Then a per-segment isotonic map (PAVA, order-preserving) on each cell’s own un-downsampled, offset-corrected holdout. Collapses ~2× COPC spread to [0.98, 1.03], dispersion ~17× smaller.

Gotchas: forgetting the offset (invisible until revenue drops), double-correcting, and a segment-varying w under a global offset (manufactures the expensive segment error). Downsampling correction is exact only if negatives are dropped uniformly at random.

Model size from latency (before quality)

20 ms is a residual of the 100 ms wall clock, not a choice.
FLOP budget    = 0.020 s × 60 GFLOP/s = 1.2 GFLOP/request
per candidate  = 1.2 GFLOP / 100      = 12 MFLOP
params ceiling = 12 MFLOP / 2 FLOP    = 6M parameters, and not one more

• Ships at 1.45M params = 2.91 MFLOP/candidate = 4.85 ms for 100 (¼ of budget). A 100M model is 333 ms = 17× over a hard external deadline (unbuildable here).
• Indexed vs traversed: the 1.15 GB embedding table is read at ~288 KB/request (indexed → free); the 5.8 MB MLP is traversed 100% every batch. “Make the model bigger” = make the embedding bigger.
• Batch the 100 into one matrix multiply, else weights stream from memory 100× and the system is memory-bound.

Features and failure modes

• High-cardinality categoricals → embeddings (16 dims). Don’t embed user id (128 GB, 86% of footprint; median user has 0 clicks so the vector decays to the prior) — use aggregate history features. Hash the rest into 2^24 buckets; collisions land in the Zipfian tail where they don’t matter (only 2.9% of head ads collide).
• The click log is not a sample of the world — only shown ads, only in their slots, only with arrived outcomes, only against gamed creatives:

• AUC is structurally blind to all of this (invariant to every monotone transform). Report RIG, gate on COPC dispersion. Use grouped (per-auction) AUC 0.594, not pooled 0.812.