07 — Event Recommendation

“Recommend local events our users might want to attend — concerts, meetups, classes, markets.”

Local events are items that have no history and never will — and building a recommender for them is a lesson in how to reason about any problem where the thing you recommend expires.

By the end you should be able to explain, out loud, four things:

• why the standard recommender architecture is structurally wrong here,
• which features replace it,
• why the label you most want arrives nine days late,
• how a geographic constraint replaces a similarity search entirely.

Everything below is derived from numbers you can check yourself.

This chapter stands on its own. Two ideas from elsewhere in this repo show up below, and both are restated in place before they are needed:

• how ranking quality is scored (Ranking and recommendation metrics),
• why gradient-boosted decision trees still beat neural networks on spreadsheet-shaped data (Why gbdts still beat neural nets on tabular data).

Follow the links for the full derivations. You do not need them to follow this chapter.

What goes in and what comes out. In goes one user — their location right now, their handful of past actions, their friend list, and the current time — together with the live catalogue of every event happening anywhere.

Out come 25 events, ranked, for the home feed, plus a smaller “near you this weekend” module.

One user in, twenty-five ordered event ids out, in under 250 milliseconds.

One word to fix now. RSVP means the button a user presses to say they intend to go. It is free, it is instant, and it is not the same thing as showing up. That gap is the central trap of the chapter.

The hard label problem, stated before anything else

Every recommender you have seen assumes an item you can learn about. A movie accumulates ratings for a decade. A product accumulates purchases forever. An event exists once, at one time and one place, has no interaction history before it happens, and is worthless the moment it is over.

Cold start is the standard name for the situation where a system has to make a recommendation about an item it has no data on — a new film, a just-listed product. In every other recommender it is an edge case that decays: wait a week and the item warms up. Here it does not decay, because the item is destroyed before it warms up. Cold start is not the exception in this system; it is the normal case, and it is permanent.

That single fact disqualifies the default architecture, and saying it in the first minute is the difference between a design and a recitation.

First thing to say: “Cold start is not an edge case here, it is the entire problem. Every item in the catalog is permanently cold, because the interaction history an item accumulates peaks at exactly the moment its value hits zero. So collaborative signal cannot carry the ranking — content and context have to, and the two context features that carry the most are where the user is and when they are free.”

The system, model by model

Here is every learned or fitted component in the finished system, listed before any derivation. The point is that when a later section says “the attendance head,” you already know what it eats and what it emits.

Notice that three rows say “Not a model.” That is deliberate. Half the good decisions in this chapter are decisions not to learn something.

Four terms in the table need defining first:

• Offline — it runs on a schedule and its output is stored.
• Online — it runs inside the 250 ms of a live feed request.
• NDCG@25 — normalized discounted cumulative gain over 25 slots. It scores an ordering: reward each relevant item, discount it by how far down the list it sits, add those up, then divide by the best score any ordering of the same items could reach. So 1.0 is a perfect ordering. Offline gives the graded version used here.
• Calibrated — a probability that means what it says. Events scored 0.30 are attended about 30% of the time. That is what lets you compare the number against a threshold instead of only against other scores (The ml objective).

Read the table one row at a time. The two columns that carry the argument are “Where its labels come from” and “Offline / online.”

Read the “where its labels come from” column top to bottom and the chapter’s whole shape is visible. Two components have no labels at all, one has labels that arrive instantly and mean the wrong thing, and one has labels that mean the right thing and arrive nine days late. There is no row anywhere in that table whose labels come from the item’s own history, because — as Why collaborative filtering structurally fails here derives — that history does not exist when you need it.

1. Framing

Before any modelling, pin down the scale, the deadline, and the five questions whose answers actually move the design — so that everything after this is arithmetic rather than opinion.

Two shorthands in the table below. MAU is monthly active users, the count of distinct people who used the product at least once in the last 30 days. p99 250 ms means the slowest 1% of feed requests must still finish inside 250 milliseconds — the 99th percentile, which is the number a latency budget is actually written against, because averages hide the requests that lose users.

Five clarifying questions are worth asking at the start. Each one has an answer that visibly changes the design rather than merely colouring it in.

One acronym first. ANN in the second row is approximate nearest neighbour search: the family of index structures that finds items whose vectors point in roughly the same direction as a query’s, trading exactness for speed. It is the reflex answer to “how do you narrow three million items down.” Model choice is where this system declines to use one.

2. Why collaborative filtering structurally fails here

The default recommender architecture cannot work here, and the proof is nothing more than counting parameters against observations. One collaborative signal survives anyway, and finding it is the derivation the whole chapter is built on.

What collaborative filtering is, and how it is usually built

Collaborative filtering is the idea that powers most recommenders: recommend to you what people who behaved like you also liked. It learns purely from the table of who interacted with what. It understands nothing about the items themselves — not their titles, not their prices, not where they are.

Its standard implementation is matrix factorization. Represent each user and each item by a short list of numbers — a vector — and predict how much a user will like an item by multiplying their two vectors together. Nobody chooses those numbers by hand. They are fitted so the products reproduce the interactions actually observed.

The length of each vector is d, called the latent dimension. d = 64 is typical.

The counting argument

The problem is arithmetic, not opinion.

An item’s vector q_i holds d free numbers that all have to be fitted. Each observed interaction with that item supplies roughly one number’s worth of evidence. So you are solving for d unknowns using however many interactions you have.

When you have fewer observations than free parameters, the fit is underdetermined: infinitely many vectors explain the data equally well. Which one you land on is then decided by the regularizer — the penalty term the fitting procedure adds to pull unconstrained numbers toward a default. That default is usually the average behaviour across all items.

The working rule of thumb: you need on the order of 10 · d observations before the fitted vector describes the item rather than the regularizer.

d = 64                              typical latent dimension
observations needed  ≈  10 · d  =  640 interactions per event

So the bar is 640 interactions. Now measure what an event actually has, at the moment a recommendation is worth making. The second number is the one to look at — not the lifetime total, but the total at the point a recommendation still has value:

median event, total RSVPs over its whole life                34
RSVPs accumulated by 7 days before the event (peak value)     9

free parameters in q_i                                        64
observations available                                         9
                                                        ----------
64 / 9  =  7.1
system is underdetermined by 7.1x

You are asking for 64 numbers and handing over 9 facts.

An aside: sanity-checking the “34 RSVPs per event”

The 34 is load-bearing, so check it against the platform totals before leaning on it. The obvious check appears to fail by a factor of 5.5, and understanding why is worth a minute.

The check that seems to fail. If 3 million events are live at any moment and each one lives 17 days, then events must be created at:

3,000,000 live  /  17 days  =  176,000 created per day
176,000 x 34 RSVPs each     =  6.0 M RSVPs per day

But Data and labels three targets and a nine day delay reports only 1.1 M RSVPs a day. That is 5.5x too many.

The error is in the check, not in the 34. The step “3 M live / 17 days = 176,000 per day” is Little’s Law: in a system at steady state, inventory = arrival rate × average time in system. The word that matters is average. The 17 days is a median — half of events are listed for less than 17 days, half for more — and the two are not interchangeable when the distribution is skewed.

And this distribution is heavily right-skewed. A trivia night gets listed on Tuesday for Thursday. A summer festival or a conference gets listed months ahead. A long right tail pulls the mean far above the median.

So back the mean out of the data instead of assuming it equals the median. Go the other direction — start from the RSVP volume, which is measured:

creation rate  =  RSVPs/day / RSVPs per event  =  1.1 M / 34   =  32,400 events/day
mean lifetime  =  live inventory / creation rate =  3 M / 32,400 =  93 days
                                                              --------
mean 93 days against a median of 17                             5.5x skew
check: 32,400 x 34  =  1.10 M RSVPs/day        closes against §4
cross-check: 32,400 x 7 = 227 k new events/week, against the ~200 k/week
             the §10 cost table already pays content-tower inference on

Both check lines close, so the 34 stands and the mean lifetime is 93 days.

Median lifetime is the number that describes an event; mean lifetime is the number Little’s Law needs; here they are 5.5x apart. Nothing in Scale and cost moves, because every storage line is built on the live inventory of 3 M rather than on a creation rate. But any per-day creation figure read off the median would have been wrong by 5.5x — and that is the kind of error that survives review, because both inputs are quoted correctly and only their combination is wrong.

What an underdetermined fit actually returns

Back to the main argument. A common misreading is that too little data gives you a noisy item vector. It does not.

An underdetermined factorization returns the regularizer’s prior mean — the default the penalty pulls toward when the data says nothing. That default is the global popularity term, with no item identity in it at all. Here the data says nothing 64 times over.

So you have not built personalization. You have built a popularity ranker carrying 64 wasted numbers per row.

And here is the fraction of the catalog that could ever clear the 640-interaction bar:

events reaching 640 lifetime RSVPs                          0.3 %
                    ... and they reach it AFTER they happen

An ID embedding is a vector attached to nothing but the item’s identifier, learned purely from that item’s own interactions. It is the thing 99.7% of this catalog can never support. The 0.3% that can, reach the threshold only after the vector has become useless.

The timing is what kills it

The shape of the problem is easiest to see as a timeline. Follow the median event from creation to the day after it happens, and watch the two right-hand columns move in opposite directions:

Information about the item and value of the item are anti-correlated in time — as one rises, the other falls. That is the sentence to have ready in an interview.

In movie recommendation the two move together: the longer an item lives, the better you know it and the more people it can still serve. Here they trade against each other.

Side by side, so the difference is not just an assertion:

The one collaborative signal that survives

One collaborative signal does survive, and volunteering it unprompted is worth points: the user’s social graph crossed with the event’s current RSVP list.

Why it survives: it is not a property of the event’s history. It is a property of who has already said yes, and that exists from the very first RSVP.

The numbers below are lift — the factor by which this user’s chance of going rises relative to their own baseline rate. So 4.7x is not a probability. It is a multiplier on one:

0 friends attending      1.0x   (baseline)
1 friend  attending      4.7x
2 friends attending      7.9x
3+ friends attending    11.2x

The strongest single feature in the system is a join between two tables that both exist on day one. It is collaborative filtering that does not need the item to have a history, because the “collaboration” is over people you already know about.

3. The ML objective

The ML objective — the exact quantity the machine-learning system is being asked to be good at — fits on one line here, and three design decisions hide inside that line of notation. The third is the one candidates almost never justify.

The model does pointwise binary classification: it looks at one (user, event) pair at a time, in isolation, and answers a yes-or-no question about it with a probability. The feed is then sorted by that probability.

p  =  P(user u attends event e | u attends nothing else conflicting, u sees e)

Read the vertical bar as “given that”. So: the probability that user u attends event e, given that nothing else on their calendar clashes and given that we actually showed it to them. Three things are worth saying about that expression, because each is a design decision hiding in the notation.

1. The target is attendance, not RSVP. They are different events with different rates (Data and labels three targets and a nine day delay), and optimizing the wrong one is a measurable, directional failure that looks like a win on the dashboard.

2. | u sees e is not free. An impression is one event appearing in one user’s feed — the unit of “we showed it”. Training data consists only of logged impressions from the current ranker, so the model is fit on the distribution the current ranker chose to produce, and it never sees anything that ranker refuses to show. Standard, and it is the Why offline ranking metrics disagree with online ctr feedback loop; the mitigation is an exploration slot (Two sided exposure and the new organizer trap), not a clever loss.

3. Pointwise, not pairwise, and this is the non-obvious one.

The alternatives have names. A pairwise loss trains on “is A better than B” comparisons. A listwise loss trains on whole orderings.

Both of them learn an order and throw the scale away. To a pairwise loss, a model that scores every event at 0.9 and a model that scores every event at 0.02 are identical, as long as the ranking matches.

Here you need the scale, for two reasons.

• The feed has a quality floor. Showing 25 events when only 4 are any good is worse than showing 4. Deciding that requires knowing whether 0.9 or 0.02 is the actual number.
• The notification path fires on an absolute threshold. That threshold is set by weighing the cost of interrupting someone against the value of them going — which only works if the score is a real probability.

A ranking loss cannot answer “is anything here worth a push notification,” and that is half the product.

So the recipe is three steps: fit a pointwise logistic loss (the standard loss for probability-valued yes/no predictions), then rank by the calibrated score (adjusted so events scored 0.30 really are attended about 30% of the time), then threshold that score against a cost matrix (Choosing a threshold from the cost matrix).

4. Data and labels: three targets and a nine-day delay

Here is the label problem promised at the top, in full. Four outcomes are observable; the useful one arrives nine days after the decision it was supposed to judge, and the fast substitute for it turns out to be actively harmful — a conflict one decomposition resolves.

Four observable outcomes, in increasing order of value and decreasing order of availability. A geofence ping in the third row is a signal from the phone that the user physically entered a circle drawn around the venue, which is how attendance gets observed for events with no ticket and no door scan.

4.1 The label-delay problem, stated as a timeline

t - 17 d   organizer creates the event
t -  7 d   recommendation served              <- the row whose label we want
t -  7 d   RSVP fires (or does not)
t -  0     event happens; attendance observed
t +  2 d   rating survey closes
                                              -------
complete label available at   t + 2 d  =  9 days after the recommendation

A model retrained nightly on complete labels is always fitting a nine-day-old world, and no amount of pipeline engineering fixes that — the delay is physics, not latency. The event genuinely has not happened yet; there is no faster database. Three consequences follow, and interviewers probe all three:

1. You cannot gate a daily A/B on the real metric. The experiment’s readout lags its exposure by 9 days minimum, and by 30+ days if you want an attendance-per-user window (Metrics).
2. Seasonal drift is discovered late. A distribution shift — the world changing so that what worked last month stops working — that begins on a Friday is not visible in complete labels until the Sunday nine days later.
3. You will be tempted to train on RSVP because it is available now. Do not, or at least not naively — see the next subsection.

4.2 The RSVP-to-attendance funnel is not uniform, and that is the trap

Across the whole platform, 62% of RSVPs turn into attendance:

overall  P(attend | RSVP)  =  0.62

That single number is a trap. Split it by the properties of the event and by who is going, and it falls apart. Look at the first two rows especially — free versus paid is a more-than-2x swing:

Read those slices together and the mechanism is obvious. An RSVP has no price. A ticket does, and so does a long drive.

Free and distant events are systematically over-RSVPed and under-attended. So a model trained on RSVP is trained to find exactly the events people will flake on.

Pricing the damage

The damage is not hypothetical. You can price it. Write f for the fraction of recommendations that are free events.

Step 1: work out the current mix. Do not assume it. The overall 0.62 is a weighted average of the free rate (0.41) and the paid rate (0.88), so it pins f down:

0.41 f  +  0.88 (1 - f)  =  0.62
0.88  -  0.47 f          =  0.62
0.47 f                   =  0.26
f                        =  0.553

So the current recommendation mix is 55% free, 45% paid.

Step 2: assume an RSVP-optimized model shifts that mix. Trained on RSVPs, it favours the free events people RSVP to freely. Say it moves the mix to 72% free / 28% paid. The new attendance-per-RSVP is the same weighted average with the new weights:

0.72 x 0.41  +  0.28 x 0.88   =   0.2952 + 0.2464   =   0.5416

Step 3: combine the win and the loss. The model does get better at its own objective — RSVPs rise 9%. But each RSVP is now worth less:

attendance, relative to before  =  (RSVPs up 9%) x (each worth 0.5416 instead of 0.62)
                                =  1.09 x 0.5416 / 0.62
                                =  0.590 / 0.62
                                =  0.952
                                   -------
RSVPs  +9 %   ·   attendance  -4.8 %

The model got measurably better at the metric it was trained on and 4.8% worse at the thing the business sells. That trace is the single most useful thing to have ready for this chapter.

One step in it is worth defending, because it is where the argument would otherwise fall apart. Step 1 derives the free/paid mix from the stated 0.62 rather than assuming 50/50.

Why that matters: an even 50/50 mix implies a baseline of 0.5 × 0.41 + 0.5 × 0.88 = 0.645, which is 4% above the measured 0.62. Run step 3 against 0.645 and the damage reads as 8.5% instead of 4.8% — nearly double, and inflated by an assumption you made up. Derived, the trap is real at 4.8% and stays real under scrutiny.

4.3 The fix: two heads, one trained late

Train a single model with two outputs and combine them:

score  =  p_rsvp(u, e)  ·  p_attend_given_rsvp(u, e)

Two words carry this. The marginal probability of an RSVP is its overall rate, ignoring everything downstream. The conditional probability of attending given an RSVP is the follow-through rate among people who already said yes.

The first head, p_rsvp, predicts the marginal. It retrains hourly on same-day labels, so it tracks the fast-moving catalogue.

The second head, p_attend_given_rsvp, predicts the conditional. It retrains weekly on labels that are nine days old, and that staleness is fine — which is the whole point of the split. The mix of events on the platform changes hourly. But the probability that a person who RSVPed to a free event 20 miles away actually turns up does not change from week to week.

One caveat on the second head, and it is one level deeper than the obvious one. It inherits the same | u sees e selection bias as the first head. Its training population is not RSVPs — it is RSVPs the current ranker produced. So it knows nothing about follow-through on events this ranker declines to show. The ml objective is loud about that for p_rsvp and silent here. The mitigation is the same exploration slot, and it is why the randomized half of slot 25 has to be logged all the way through to attendance rather than only to RSVP.

Split a delayed label into a fast marginal and a slow conditional, and only the fast part needs fresh data. That decomposition is the answer to “how do you handle delayed labels” in this chapter, and Delayed conversions and the bias they inject runs a harder version of the same problem.

Here it is in code. There is not much to it — the point of writing it down is to guard against anything else creeping into the product. Watch the two printed rows: the second candidate has the higher RSVP probability and still loses:

def event_score(p_rsvp, p_attend_given_rsvp):
    """Section 4.3: the ranking score IS the probability of attendance.

    Two heads, multiplied, and nothing else. Distance already enters through
    both heads as a feature, so a post-hoc distance multiplier would double
    count it and -- worse -- decalibrate the output, which section 8's
    notification threshold reads as an absolute probability.
    """
    return p_rsvp * p_attend_given_rsvp


print("%-26s %8s %11s %9s" % ("candidate", "p_rsvp", "p_att|rsvp", "score"))
for _name, _r, _a in (("paid, 2 mi, 2 friends", 0.20, 0.62),
                      ("free, 20 mi, alone", 0.26, 0.44)):
    print("%-26s %8.2f %11.2f %9.4f" % (_name, _r, _a, event_score(_r, _a)))

assert abs(event_score(0.20, 0.62) - 0.124) < 1e-12
# the free-and-distant candidate has the higher RSVP probability and the lower
# joint -- section 4.2's whole trap, in two rows
assert event_score(0.26, 0.44) < event_score(0.20, 0.62)

candidate                    p_rsvp  p_att|rsvp     score
paid, 2 mi, 2 friends          0.20        0.62    0.1240
free, 20 mi, alone             0.26        0.44    0.1144

4.4 Negative sampling

A classifier needs examples of both answers — which means working out where the “no” examples come from, and how rare the “yes” ones are.

Negative sampling is the choice of which non-events to show the model as zeros. Get it wrong and the model learns the wrong contrast.

How rare is a positive?

Impressions that produced no RSVP are the natural negatives. Compute their rate from the volume table rather than guessing. The request figure comes from Scale and cost, the RSVP figure from the table above, and the 15,000 candidates from Why the geo index is the whole retrieval story:

requests/day     240 M/week / 7            =  34.3 M      (§10)
impressions/day  34.3 M x 25 slots         =  857 M
RSVPs/day                                  =  1.1 M       (§4)

positive rate, impression level   1.1 / 857     =  0.128 %
positive rate, (user, candidate)  1.1 / 514,500 =  0.00021 %  (15,000 cands, §9.1)

Two readings of that, both worth saying out loud:

• 0.032 RSVPs per request. 1.1 M / 34.3 M = 0.032, so one RSVP per 31 feed loads — not one per feed load.
• 780 negatives per positive at the impression level, since 1 / 0.00128 = 780.

Which negatives to use

Use impressed-but-not-RSVPed as the primary negatives. Then mix in 20% random candidates that were never impressed at all.

The random ones do a job impressions cannot. They teach the model that Tuesday-morning knitting in the next county is not a match. Impressions alone can never teach that, because the current ranker never shows those events and so never produces a row about them.

The correction that injection forces

Injecting those rows carries the same correction obligation that Training puts on downsampling, run in the opposite direction — and it is the easier one to forget, because downsampling visibly deletes rows while injection feels like it is adding realism.

The mechanism: adding never-impressed rows as hard zeros adds negative mass and no positive mass. So the marginal positive rate drops by the share injected, and every predicted probability drops with it.

The code below fits the distortion and then undoes it. The key detail is in the uncorrected docstring: the distortion is a constant factor on the odds, not on the probability. Odds here means p / (1 - p) — the probability expressed as a ratio of yes to no. Applying the factor to the probability directly gets a different, wrong answer, and the docstring gives the size of that mistake:

IMPRESSION_RATE = 1.1e6 / 857e6          # 0.128 %, section 4.4


def injected_rate(p, inject_share):
    """Marginal positive rate after injecting never-impressed rows as zeros."""
    return p * (1.0 - inject_share)


def uncorrected(p_true, p_base, inject_share):
    """What a model fit on the injected mix emits for a true probability.

    The distortion is a constant factor on the ODDS, not on the probability:
    injection multiplies the negative mass and leaves the positive mass alone,
    which is the negative-downsampling correction of section 7 run backwards.
    Applying it to the probability instead understates a true 0.30 as 0.240
    rather than 0.255 -- wrong in the same direction, and wrong by a third of
    the error it is trying to remove.
    """
    ratio = (1.0 - inject_share) * (1.0 - p_base) / (1.0 - p_base * (1.0 - inject_share))
    odds = p_true / (1.0 - p_true) * ratio
    return odds / (1.0 + odds)


def correct(p_emitted, p_base, inject_share):
    """Undo it. This is the line that must exist somewhere in the pipeline."""
    ratio = (1.0 - inject_share) * (1.0 - p_base) / (1.0 - p_base * (1.0 - inject_share))
    odds = p_emitted / (1.0 - p_emitted) / ratio
    return odds / (1.0 + odds)


print("inject   marginal rate      a true 0.30 is emitted as")
for _f in (0.05, 0.10, 0.20):
    print("  %2d%%    %.4f%% -> %.4f%%   x%.3f        %.4f"
          % (_f * 100, IMPRESSION_RATE * 100,
             injected_rate(IMPRESSION_RATE, _f) * 100, 1.0 - _f,
             uncorrected(0.30, IMPRESSION_RATE, _f)))
    assert abs(injected_rate(IMPRESSION_RATE, _f)
               - IMPRESSION_RATE * (1.0 - _f)) < 1e-15
    # the correction is exact, which is the whole reason to prefer it to
    # re-fitting calibration on a distribution you deliberately distorted
    assert abs(correct(uncorrected(0.30, IMPRESSION_RATE, _f),
                       IMPRESSION_RATE, _f) - 0.30) < 1e-12
assert uncorrected(0.30, IMPRESSION_RATE, 0.20) < 0.26      # ~15 % relative, low
assert injected_rate(IMPRESSION_RATE, 0.0) == IMPRESSION_RATE

inject   marginal rate      a true 0.30 is emitted as
   5%    0.1284% -> 0.1219%   x0.950        0.2893
  10%    0.1284% -> 0.1155%   x0.900        0.2783
  20%    0.1284% -> 0.1027%   x0.800        0.2553

Read the last row. At the 20% injection this section just prescribed, a true 0.30 comes out as 0.255. That is a 15% relative miscalibration, landing on exactly the quantity Metrics’s notification threshold consumes.

Injection is still the right call, because the never-impressed rows carry information no impression can. What is not optional is the shift back.

Downsample and correct, inject and correct. The two are the same identity with the sign flipped, and a pipeline that does one and not the other is miscalibrated by construction.

5. Features: geography and time carry the load

With no item history to lean on, the features are the model. Geography comes first, where the obvious encoding is provably unlearnable and the fix is one precomputed number; then time, where the textbook technique is the wrong choice and raw memorization wins; then the rest of the vector, counted out so that “a tree model over engineered features” becomes an actual specification.

5.1 Raw latitude and longitude is a bad feature, and here is why

How much does a decision tree waste when you hand it raw coordinates? The answer can be quantified exactly, and it is one of the cleanest arguments in the chapter.

Start with what a tree can express. A decision tree makes predictions by asking a chain of yes/no questions. Each question is a split, comparing one feature against a threshold. The answer at the end of a chain is stored in a leaf. So every split cuts along one axis at a time — vertical or horizontal, never diagonal, never curved.

Now what the target needs. Attendance depends on the difference between two positions, not on either one by itself. But a tree can only split on things like lat_event < 37.77, which is a statement about absolute position.

So “within 5 miles of the user” — a disc — is not expressible as any single split. The tree has to re-learn it as a separate box-shaped region, once for every distinct user location.

The damage can be quantified. Approximate a disc of radius r (the user’s neighbourhood) by a staircase of k axis-aligned bands, which is the best a depth-limited tree can do. The question is how much of the disc that staircase covers:

k bands, each INSCRIBED (width set at the band's far edge, so the staircase
stays strictly inside the disc)

k bands     area covered / area of disc
   4              55.1 %
   8              79.4 %
  16              90.4 %
  32              95.4 %
  64              97.8 %

The word inscribed is load-bearing, and it is the only convention under which those five numbers are correct. It means each band’s width is set at whichever of its two edges is further from the centre, so the staircase stays strictly inside the disc and never over-claims.

Take each band’s width at its midpoint instead — the other obvious rule — and the staircase spills outside the disc. That gives 103.7% coverage at k = 4, which is nonsense on its face. A reader who quietly picks the other convention will spend an afternoon wondering why their numbers do not match, which is why the convention is printed inside the block.

Now the punchline, and mind the units. Sixteen bands capture 90% of one disc. A band is not a leaf: each band costs one y-split plus two x-splits, so sixteen bands is about 48 leaves. The table is headed k bands, not k leaves, and that difference is a factor of three against the design being argued for.

And the tree does not get to reuse that structure. A user in Oakland needs a different forty-eight leaves than a user in San Jose. So the leaf budget scales with the number of distinct user neighbourhoods, and the feature is effectively unlearnable.

The fix is one precomputed number:

haversine(user, event)  ->  one scalar
"within 5 miles"        ->  ONE split

Haversine distance is the great-circle distance between two latitude-longitude points — how far apart they are measured over the curved surface of the earth rather than straight through it. It collapses two positions into one number, and 48 leaves into 1 split.

Compute the difference before the model sees it. That is the general rule for any translation-invariant relationship — one where shifting both things by the same amount changes nothing, so only the gap matters. It is the same argument as Crosses and interactions: if the target is a function of an interaction between two features, hand the model the interaction rather than the two features.

5.2 What to use instead

Six geographic features replace the two coordinates. Choosing between them is mostly a question of which map grid resolution each job needs.

Four terms before the table:

• H3 — a global grid that tiles the earth in hexagons at several resolutions. Each cell has a single integer id. Higher resolution number means smaller cells.
• Categorical — a feature whose values are labels rather than magnitudes. An H3 cell id is a label: cell 8837 is not “bigger” than cell 8836.
• Cardinality — how many distinct values a feature can take.
• Target encoding (TE) — how a tree consumes a high-cardinality categorical. Replace the label with the historical outcome rate observed in that cell, so the cell id becomes a number the tree can split on.

Three resolutions, three jobs

The system uses three H3 resolutions for three different jobs. Conflating them is the single most common way this design goes wrong.

Here is what that mistake costs. Size a ring in res-5 cells but take it at res 7, and it covers about 7 km instead of about 48 — a seven-fold under-fetch. Exactly seven, because a res-7 edge is a res-5 edge divided by 7. Nothing anywhere in the pipeline reports it as an error; the feed just quietly gets smaller.

The “Sized in” column says which section fixes each resolution:

The logic behind those three choices is one line each:

• Coarse cells for crosses. A cross of two categoricals multiplies their cardinalities. Fine cells would produce far more combinations than there is data to fill.
• Medium cells for rings. Each cell in the ring costs one index lookup, so more cells means more seeks (Why the geo index is the whole retrieval story).
• Fine cells for travel time. About 5 km is the scale at which a single routing estimate is still honest about a whole neighbourhood.

Two further notes on the encoding are worth volunteering unprompted, because both are about H3 being the right shape of grid and not just a convenient one:

• Geohash cells are not equal-area. A geohash is the older, square-celled alternative to H3, which encodes a position by repeatedly halving the latitude and longitude ranges. A precision-5 geohash is about 4.9 km × 4.9 km at the equator and about 2.4 km wide at 60 degrees latitude, because longitude degrees shrink with the cosine of latitude. That is a silent bug: a “same cell” feature means something different in Oslo than in Nairobi. H3’s hexagons are near-equal-area and every neighbor is equidistant, which also makes ring queries clean.
• Hexagons have one neighbor distance; squares have two. A square cell’s diagonal neighbor is 1.41x further than its edge neighbor, so “adjacent cell” is an inconsistent unit. Hexagons do not have this problem, which matters when the ring is the retrieval primitive (Serving).

5.3 Distance decays, but not according to any law

Willingness to travel falls off with distance — but in a shape no standard curve reproduces. The right response is to replace the curve with buckets and, where affordable, to replace distance itself with something better.

Here are the measurements: relative RSVP rates against distance, normalized so the 0-2 mile band reads 1.00. By 40 miles the rate is a hundredth of what it is next door:

miles      0-2    2-5    5-10   10-20   20-40   40+
rel rate   1.00   0.61   0.33   0.14    0.045   0.011

Two standard shapes are worth trying against that:

• Exponential decay, exp(-d/tau) — falls by a constant fraction for every fixed distance added. The parameter tau is the distance over which it falls to about a third.
• Power law, d^-a — falls by a constant fraction for every fixed multiplication of distance.

The test is simple. If either shape were the right one, fitting its parameter to each adjacent pair of bands would give the same answer every time. A real constant does not drift.

But first: the table above is a set of bands, not a set of distances. Nothing can be fitted to it until you say what single distance each band is worth. State that convention before quoting any parameter, because a reader who picks a different one gets different numbers and no way to tell which of you is wrong:

each band taken at its GEOMETRIC midpoint sqrt(lo*hi), with the 0-2 band at
its arithmetic midpoint (sqrt(0*2) is 0) and 40+ read as 40-80:

    d (mi)   1.0    3.16   7.07   14.14   28.28   56.57
    rel rate 1.00   0.61   0.33   0.14     0.045   0.011

exponential  exp(-d/tau):  implied tau = 4.4, 6.4, 8.2, 12.5, 20.1   -> 4.6x spread
power law    d^-a:         implied a   = 0.43, 0.76, 1.24, 1.64, 2.03 -> 4.7x spread

Neither parameter is remotely constant. The implied tau runs from 4.4 to 20.1 — a 4.6x spread, not the 1.9x that a tau quoted as “7.1 to 13.3” would suggest. So the argument for abandoning the curve is about two and a half times stronger than the usual telling of it.

Change the band-to-distance convention — arithmetic midpoints, geometric midpoints, upper edges, lower edges, any reading of the open 40+ band — and the qualitative result holds while the exact numbers move. That is precisely why you publish the convention next to the numbers.

So why does neither form fit? Because this is not one decay. It is a mixture of transport modes, with thresholds between them:

• Under about 2 miles you walk.
• Between 2 and 10 you drive or take transit, and the cost is your evening.
• Past 20 miles the event has to be worth a trip.

The kinks in the curve sit exactly where the mode changes — and the mode boundaries move from metro to metro, so there is not even one shared set of kinks to fit.

So do not fit a curve at all. Do two things instead.

1. Bucket the distance and let the GBDT learn the step function directly. Six buckets, with boundaries at 2/5/10/20/40 miles, which is where the kinks are.
2. Replace distance with travel time wherever you can afford it, because travel time is the quantity that is actually comparable across cities:

8 miles, New York, 7pm Friday, transit         35 min
8 miles, Los Angeles, 2pm Tuesday, driving     22 min
8 miles, Los Angeles, 6pm Friday, driving      55 min
                                              --------
same 8 miles, 2.5x range in the thing that matters

Travel time cannot be computed live, one route at a time. QPS is queries per second, the rate of incoming requests. Do the multiplication:

1,200 QPS peak  x  15,000 candidates  =  18,000,000 route calculations per second
  397 QPS avg   x  15,000 candidates  =   6,000,000 per second

No routing service answers 18 million calls a second, and capacity is always sized for peak rather than average.

Precompute instead, as a matrix of travel times from every map cell to every other map cell. Two terms in the sizing block: a daypart is one of four coarse time-of-day slices, because traffic at 8am and at 2pm are different worlds; the two modes are driving and transit. Follow the multiplications down to the last line:

metro area                       ~5,000 km^2
H3 res 7 cell area                 5.16 km^2   ->  ~970 populated cells
cell pairs                         970^2       =  941,000
x 4 dayparts x 2 modes                         =  7.5 M entries
x 4 bytes                                      =  30 MB per metro
x 200 metros                                   =  6 GB, rebuilt nightly

Six gigabytes turns a per-request routing call into an array lookup. That is the whole trick, and it is the kind of number an interviewer is checking you will produce rather than hand-wave.

5.4 Time, and why you should memorize the calendar rather than model it

Time is the second feature carrying real weight, and the argument here runs against the textbook: the standard encoding is a tool for when you are short of data, and here you are not — though there is a place where you are, and it needs different handling.

The textbook fix, and why to skip it

Hour-of-day encoded as an integer from 0 to 23 is wrong for any model that treats the feature as a magnitude. Hour 23 and hour 0 are one hour apart in reality and 23 units apart numerically.

The textbook fix is cyclic encoding: represent the hour by its position on a circle, using sine and cosine, so 23 and 0 land next to each other.

Two terms in the block below. Each sine-cosine pair is a harmonic, and adding more harmonics lets the fitted curve have more peaks per day. Degrees of freedom counts how many independent numbers the resulting shape has to play with — how flexible it is.

sin(2 pi h / 24),  cos(2 pi h / 24)          one harmonic  -> one peak per day
+ sin(4 pi h / 24), cos(4 pi h / 24)         two harmonics -> lunch and dinner
k harmonics represent any daily shape with 2k + 1 degrees of freedom

But you should usually skip it here, because you have enough data to memorize the calendar instead.

The quantity that matters is not hour-of-day and not day-of-week. It is their interaction: Friday 8pm and Tuesday 8pm are different products. So bucket on hour-of-week and count what you have:

hour-of-week buckets                     7 x 24  =  168
historical RSVPs                                 =  200 M
per bucket                          200 M / 168  =  1.2 M

With 1.2 million observations in every one of the 168 buckets, a fitted harmonic cannot beat simply averaging what happened in that bucket. So use the 168-way bucket.

Cyclic encoding is a smoothing prior — an assumption imposed to fill in for missing evidence — and priors are for when you are short of data. Here you are not.

Where you are short of data

The interaction that actually matters is metro by category by hour-of-week. Salsa night in Miami on a Friday is a different market from ceramics in Portland on a Tuesday. Slice that finely and the counts collapse:

metro x category x hour-of-week   =  200 x 30 x 168  =  1.0 M cells
200 M RSVPs / 1.0 M cells         =  200 RSVPs per cell   <- a NUMERATOR

An aside on units: 200 of what?

That last line has a trap in it, and it is worth two paragraphs because the whole shrinkage argument reads off it.

Two hundred RSVPs is a numerator. The shrinkage formula n / (n + m) wants a denominator — a trial count. RSVPs are the successes; trials are the impressions those successes came out of. Say which you mean before doing arithmetic with the number.

This bites here specifically because the chapter uses the word “rate” for two quantities that are ten-fold apart. Negative sampling puts the impression-level RSVP rate at 0.128%. The shrinkage argument below reads its noise off “a base rate near 0.10” — a per-cell rate on a coarser unit. Both are legitimate; they are not the same number.

Below are the four defensible readings of n = 200, and they do not agree with each other. The column to watch is “as a share of p” — the noise expressed relative to the thing being estimated. It ranges from 6.7% to 197%:

The n in n / (n + m) is the cell’s trial count, and the volume line above does not supply it. “200 M RSVPs over 1.0 M cells” fixes the numerators and says nothing at all about the denominators.

So treat the first row — 21% noise, 91% weight — as an illustration at n = 200 trials against a cell rate near 0.10, not as a measurement. It is the reading the rest of this subsection uses.

The row to be afraid of is the second one. A reader who carries the impression-level 0.128% in there gets a standard error twice the size of the estimate itself, and would conclude the level should be thrown away rather than shrunk.

That the four readings span 6.7% to 197% is exactly the argument for the prescription below: fit m per level from the ratio of within-cell to between-cell variance. That measures the noise instead of inferring it from a count whose units were never stated.

Shrinkage, and how much of its own estimate a level keeps

Two hundred observations per cell is thin enough to be noisy but not thin enough to throw away. That is what shrinkage is for: blend a sparse estimate toward a denser, coarser one, in proportion to how little evidence the sparse one has.

So shrink up a hierarchy, coarsening at each step:

cell  ->  metro x hour-of-week  ->  hour-of-week  ->  global

Two terms. A pseudocount m is a number of imaginary observations you pretend the coarser level contributed. A level keeps n / (n + m) of its own estimate and gives the rest to its parent. Empirical Bayes means you fit m from the data rather than picking it by hand.

The pseudocount has to be fitted per level, not shared across levels, because 200 observations and 3 observations need completely different amounts of help.

Watch what a single shared m = 20 does to the 200-observation cell:

weight the cell keeps  =  200 / (200 + 20)  =  0.909  =  91 %

Ninety-one percent means the shrinkage does essentially nothing — to the very level this subsection just called thin. Whether that is acceptable depends on how noisy 200 observations actually are.

So measure the noise. The sampling standard deviation is the typical amount an estimate wobbles purely because it came from a finite sample. At n = 200 trials against a base rate near 0.10:

sqrt( 0.10 x 0.90 / 200 )  =  sqrt(0.00045)  =  0.021
0.021 / 0.10               =  21 % of the rate itself

(That 21% is the first row of the table above. Read the estimand as the impression-level rate instead and the same calculation gives 197%, which is the reading the table exists to rule out.)

Twenty-one percent noise passed through at 91% weight is exactly the wobble a GBDT will happily overfit — meaning memorize as if it were signal.

The fix is the prescription already named: fit m at each level as the ratio of within-cell sampling variance to between-cell variance. Do that and the coarse levels get the larger pseudocounts they need, instead of every level sharing one number that happens to suit none of them.

Where a small m is right: the user

The same argument bites hardest on the individual user, and there a single small m is the correct answer rather than a compromise:

median user lifetime RSVPs                 3
per-user day-of-week histogram, 7 bins     3 observations over 7 bins
shrinkage toward the metro prior, m = 20:  weight on the user's own data
                                           =  3 / (3 + 20)  =  13 %

A histogram here is just the count of that user’s RSVPs falling on each of the seven days of the week. Three RSVPs spread over seven bins is almost nothing, and 3 / (3 + 20) = 13% is how much of it survives shrinkage.

The personalization you can actually afford is 13% of a day-of-week histogram. Say that number out loud in the interview. It reframes the whole problem away from “learn this user” and toward “learn this city, then nudge.”

The function that produces both numbers

Both halves — the 13% and the 91% — fall out of one function, so it is worth writing down rather than describing. Read the docstring first: the argument m is a vector, one pseudocount per shrink step, and that is the design decision the whole function exists to enforce.

The test fixtures below the function are as interesting as the function itself, and the comments explain why: the obvious test cannot fail.

def shrunk_rate(k_user, n_user, parents, m):
    """Empirical-Bayes backoff up a hierarchy of increasingly coarse priors.

    `parents` is ordered fine -> coarse, e.g.
        [(k_metro_cat_how, n_metro_cat_how), (k_metro_how, n_metro_how),
         (k_how, n_how), (k_global, n_global)]
    The coarsest entry is the root prior; every finer level is shrunk toward
    the level above it, and the user is shrunk toward the finest parent.

    `m` is ONE PSEUDOCOUNT PER SHRINK STEP, aligned with
    [user] + parents[:-1] and ordered the same fine -> coarse way. It is a
    vector and not a scalar on purpose: a level keeps n / (n + m) of its own
    estimate, so at m = 20 the user's 3 lifetime RSVPs keep 13% while a
    200-observation metro x category x hour-of-week cell keeps 91% and is,
    in practice, standing on its own. Fit each m from the ratio of
    within-cell to between-cell variance at that level.
    """
    steps = [(k_user, n_user)] + list(parents[:-1])       # fine -> coarse
    assert len(m) == len(steps)
    rate = parents[-1][0] / max(parents[-1][1], 1.0)      # root: the global mean
    for (k, n), m_level in zip(reversed(steps), reversed(m)):
        rate = (k + m_level * rate) / (n + m_level)
    return rate


print("user, 3 obs, m=20  keeps %.1f%% of its own estimate" % (100 * 3 / (3 + 20)))
print("cell, 200 trials   keeps %.1f%%" % (100 * 200 / (200 + 20)))
assert abs(3 / (3 + 20) - 0.130) < 5e-4               # the user keeps 13%
assert abs(200 / (200 + 20) - 0.909) < 5e-4           # the 200-trial cell keeps 91%

# every level sitting at a 10% rate, so the only thing moving the answer is
# the user's own 1-of-3 -- which lands at 3/23, not at 1/3
PARENTS = [(20.0, 200.0), (600.0, 6_000.0), (1.4e6, 1.4e7), (2.0e7, 2.0e8)]
M = [20.0, 20.0, 200.0, 2_000.0]
print("\nflat hierarchy, user 1 of 3   %.9f   (3/23 = %.9f)"
      % (shrunk_rate(1.0, 3.0, PARENTS, M), 3 / 23))
assert abs(shrunk_rate(1.0, 3.0, PARENTS, m=M) - 3 / 23) < 1e-9

# ...and that fixture ALONE tests almost nothing. With every level sitting at
# exactly 0.10, (k + m*0.1)/(n + m) = 0.1 for any m at all, so the hierarchy is
# a fixed point: the backoff ORDER and the per-level m -- the two things this
# function exists to implement -- are both unobservable to it.
assert abs(shrunk_rate(1.0, 3.0, list(reversed(PARENTS)), M) - 3 / 23) < 1e-9

# so move ONE number off the fixed point. The cell now sits at 20% and every
# level above it at 10%, and each way of mis-ordering the backoff shows up.
P2 = [(40.0, 200.0), (600.0, 6_000.0), (1.4e6, 1.4e7), (2.0e7, 2.0e8)]
print("cell at 20%%, parents+m fine->coarse (documented)  %.6f"
      % shrunk_rate(1.0, 3.0, P2, M))
print("             parents passed coarse->fine          %.6f"
      % shrunk_rate(1.0, 3.0, list(reversed(P2)), M))
print("             m aligned coarse->fine               %.6f"
      % shrunk_rate(1.0, 3.0, P2, list(reversed(M))))
assert abs(shrunk_rate(1.0, 3.0, P2, M) - 0.209486) < 1e-6
assert abs(shrunk_rate(1.0, 3.0, list(reversed(P2)), M) - 0.130435) < 1e-6
assert abs(shrunk_rate(1.0, 3.0, P2, list(reversed(M))) - 0.150275) < 1e-6

user, 3 obs, m=20  keeps 13.0% of its own estimate
cell, 200 trials   keeps 90.9%

flat hierarchy, user 1 of 3   0.130434783   (3/23 = 0.130434783)
cell at 20%, parents+m fine->coarse (documented)  0.209486
             parents passed coarse->fine          0.130435
             m aligned coarse->fine               0.150275

The flat fixture is kept because it isolates the 13%, and the second one is added because the flat fixture cannot fail. A test whose expected value is a fixed point of the function under test passes for every implementation that reaches a fixed point — including one that ignores the hierarchy entirely — and this is the shape that survives review most easily, because the number it checks is the number the prose quotes.

5.5 The rest of the feature set

Geography and time are the two heaviest groups, but the row the model actually scores has 134 numbers in it, and every one deserves an accounting.

Three terms recur below:

• Embedding — a short list of numbers standing in for something unstructured (a title, an image), positioned so similar things get similar lists.
• Fill rate — the fraction of an event’s capacity already taken.
• Lead time — how many days remain until the event happens.

In the table, the “Note” column is where the argument lives. Two rows to watch: Organizer, because that is where the item history from Why collaborative filtering structurally fails here actually survives, and Event state, because it is the only history the event itself has and it is two numbers rather than 64.

Counting the vector matters, because “a GBDT over engineered features” is not a specification until you can say how wide the row is. TE in the block below is the target encoding introduced in What to use instead, and how abbreviates hour-of-week:

geo (§5.2)     haversine, distance bucket, 4 daypart travel times,
               event res-7 cell TE, user home cell TE, res-4 pair TE      9
time (§5.4)    hour-of-week rate, metro x cat x how rate, week-of-year,
               days-to-nearest-holiday                                    4
social         friends attending / RSVPed / following the organizer       3
content        category, subcategory, tag count, content-tower vector    67
event state    lead time, price, capacity, fill rate, 6h velocity         5
organizer      lifetime events, attendance rate, rating, no-show rate     4
user affinity  30 category rates (shrunk), 5 price-band rates,
               mean distance travelled, past-organizer match             37
context        minutes to event, temp / precip / wind, holiday flag       5
                                                                      -----
                                                                        134

134 features, and 64 of them are one precomputed content vector — so the model the interview is actually about is 70 hand-built numbers. Every one of the 70 is a plain decimal number or a small whole number, and there is not a single learned ID embedding anywhere in the row. That is Why collaborative filtering structurally fails here restated as a database schema.

And here is one of those rows with values in it, because a count of features is not a feature vector.

This is a single (user, event) pair on its way into the ranker, with the two numbers that come out at the bottom. Two things to notice: cat_rate carries the “13% own” annotation from Time and why you should memorize the calendar rather than model it, and p_rsvp of 0.0054 is small in absolute terms but 4.2x the base rate:

user u_84213  (San Francisco, 3 lifetime RSVPs, 41 friends)
  x  event e_5590172  "Dolores Park Swing Night", Fri 3 Oct 19:00

  geo                                    time
    haversine_mi            2.4            hour_of_week          "Fri-19"
    distance_bucket         "2-5"          metro_cat_rsvp_rate    0.104
    travel_time_min        11              week_of_year          40
    event_cell_te (res 7)   0.031          days_to_holiday       13
  social                                 event state
    friends_attending       2              lead_time_days         6
    friends_rsvped          3              price_usd             15.00
    friends_follow_org      1              capacity             120
  organizer                                fill_rate              0.62
    lifetime_events        44              rsvp_velocity_6h       4
    organizer_attend_rate   0.71         user affinity
    organizer_rating        4.3            cat_rate "social/dance" 0.09 (shrunk, 13% own)
    no_show_rate            0.29           mean_distance_mi       3.1
  ... + 58 more hand-built + 64 content-tower dims  =  134

  p_rsvp                =  0.0054     (4.2x the 0.128% impression base rate:
                                       2 friends going, 2.4 mi, Friday evening)
  p_attend_given_rsvp   =  0.83       (paid $15, under 5 mi, 2+ friends going)
  score = 0.0054 x 0.83 =  0.00448    ->  rank 3 of 25

And here is the head of the list it lands in. This is where the two-head decomposition stops being notation — compare the p_rsvp column against the score column and watch the order change:

rank  event                                    p_rsvp  p_att|rsvp     score
  1   e_2210418  Bernal Heights Pub Quiz       0.0061      0.88      0.005368
  2   e_7714903  Potrero Ceramics Open Studio  0.0058      0.86      0.004988
  3   e_5590172  Dolores Park Swing Night      0.0054      0.83      0.004482
  4   e_4471903  Mission Night Market (free)   0.0072      0.58      0.004176

Slot 4 has the highest RSVP probability in the feed and finishes fourth. It is free and outdoors, so its conditional attendance rate is 0.58 against 0.88 for the paid quiz. An RSVP-trained ranker would have put it first. That is The rsvp to attendance funnel is not uniform and that is the trap’s whole trap, visible in four rows.

The absolute values look small on purpose, and they reconcile with Negative sampling. Twenty-five slots at a mean p_rsvp of 0.00128 gives 0.032 expected RSVPs per feed load. These four rows account for 0.0245 of that, which leaves the remaining 21 slots averaging 0.00036 — a steep drop-off, which is what a working ranker should produce.

The organizer is the item that has a history. An event has 9 interactions; the organizer who runs it monthly has 400. Every collaborative technique that fails on events works on organizers, series, venues, and categories — so route the collaborative machinery to those entities and let the event inherit. This is the single most reusable idea in the chapter.

6. Model choice

Choosing the models here means picking a ranker and, more importantly, declining to pick a retrieval model at all. The interesting argument is the second one, because “which approximate nearest neighbour index would you use” is the question an interviewer expects you to answer rather than refuse.

There are two stages, and the first is not a model. It is an index lookup, because the constraint that cuts three million events down to one city is geographic, and a geographic constraint has to hold exactly.

3,000,000 live events globally
   -> geo + time hard filter (§9.1)                 ->  ~15,000
   -> GBDT ranker over all 15,000                   ->  25

That 15,000 is worth deriving rather than assuming, because that number and no other decides the architecture. It falls out twice, from two independent directions — first by dividing the catalogue across metros, then by measuring the disc the product promises:

live events, globally                    3,000,000
metros                                         200
                                        ----------
live events per metro                       15,000

25-mile radius                            40.23 km
disc area  pi r^2                          5,085 km^2
metro area (§5.3)                         ~5,000 km^2
                                        ----------
the disc IS the metro          5,085 / 5,000 = 1.02x

The radius the product promises is almost exactly the size of a metro, so “events near this user” and “this city’s live catalog” turn out to be the same set.

Selectivity is the fraction of the whole catalogue that survives a filter. Here that is 15,000 / 3,000,000 = 0.50%.

The argument people expect, and the one that actually decides it

HNSW is the dominant approximate-nearest-neighbour index: a layered graph of items, where a search walks greedily from neighbour to neighbour toward the query.

The expected argument goes like this. Pre-filtered HNSW traversal — walking the graph while ignoring everything that fails the filter — degenerates into a linear scan below about 1% selectivity, because the search keeps landing on ineligible neighbours (When it is not fine say these unprompted). At 0.50% you are under that line.

Do not lean on it. You clear a rule of thumb by a factor of two, and a rule of thumb is not something to clear by 2x and then build on.

The argument that does hold weight is different: an ANN index answers “approximately the nearest,” and geo here is not a preference to be approximated. It is a predicate that must be exact. A user shown an event 40 miles outside the radius has been shown a wrong answer, not a slightly worse one. Why the geo index is the whole retrieval story prices what it costs to force exactness out of an approximate index anyway.

The consequence is that there is no ANN index in this system, and being able to say why is better than being able to name three ANN algorithms. 15,000 candidates is still small enough to score every one — but it is no longer free, and Scale and cost is where that shows up.

The ranker

Three of those rows use names worth unpacking:

• Two-tower — run the user through one network and the item through another, producing a vector each, then score the pair by multiplying the vectors. That structure is what makes it fast enough for retrieval, and also what makes it depend on the item having a learnable vector.
• MLP (multi-layer perceptron) — a plain stack of fully connected neural network layers.
• Wide & deep — a memorization component that handles enormous numbers of sparse feature combinations, sitting alongside a generalizing neural component.

The GBDT wins for the reason set out in Why gbdts still beat neural nets on tabular data. On tabular data — spreadsheet-shaped rows of hand-built numeric features, which is exactly what The rest of the feature set produced — trees still beat neural networks. And most relationships here are close to monotone, meaning the answer moves in one consistent direction as a feature rises, which is the shape trees fit most efficiently.

A neural component does earn its place in exactly one spot: a small text-and-image tower that produces the 64-number content embedding of the event, computed once when the event is created and handed to the GBDT as 64 ordinary features. That is content understanding, which trees cannot do from raw text, and it costs nothing at request time because it is precomputed per event rather than per (user, event) pair.

Scoring is not a rounding error in the latency budget. At 15,000 candidates it is the second-largest line in it. Each candidate walks 300 trees, and each tree asks 8 questions:

15,000 candidates x 300 trees x depth 8  =  36 M comparisons
vectorized LightGBM, ~1e9 comparisons/s  ->  ~36 ms

Vectorized means the library evaluates many candidates at once using wide CPU instructions rather than one at a time. That is where the billion-comparisons-per-second figure comes from.

36 ms out of a 250 ms budget makes the model the cheap part. The 15,000 feature rows you have to assemble to feed it are the expensive part — Scale and cost puts that at 28 ms, with the social join costing another 20.

The candidate count is what makes this worth saying. A design built on 4,000 candidates — a count the geometry does not support — would put scoring near 10 ms and leave the budget looking comfortable. At the 15,000 the geometry actually forces, the pipeline still fits: 138 ms of the 250 spent. But the headroom is now something you manage rather than something you have.

7. Training

Four things get fitted on four different clocks. After setting each one, most of the attention belongs to the single decision that separates a model that works from one that only looks like it works: how you split the data.

Isotonic in the last row means a fitted step function that is only allowed to go up, never down: it can change how a score maps onto a probability without ever reordering two events.

The split, which is the decision that matters

The split must be temporal. A random split here is not merely optimistic — it is a specific, nameable leak.

A leak is when information that would not exist at prediction time reaches the model during training, so the evaluation flatters it.

Here the mechanism is precise, in three steps:

1. A random split scatters impressions of the same event across both the training side and the validation side.
2. fill_rate and rsvp_velocity_6h are features, and both describe how popular that event became.
3. So a training row carries knowledge of that event’s eventual popularity, and the validation row for the same event gets scored using it.

The measured gap between the two splits, offline and online:

random 80/20 split      offline NDCG@25 = 0.412      online RSVP lift = -1.8 %
temporal split          offline NDCG@25 = 0.317      online RSVP lift = +2.4 %

The leaky split reported a 30% better model that lost online — 0.412 against 0.317 offline, and −1.8% against +2.4% where it counted.

So split by time. If you want to be strict, put the boundary before any event that appears in validation was even created.

Class imbalance: weight, or downsample and correct

Positives are rare, so the training set needs handling. Two options:

• Class weighting — tell the loss function to count each rare positive as worth many negatives.
• Resampling — physically delete negatives from the dataset.

Prefer weighting, for the reason set out in Class weights are the exact objective resampling is a noisy estimate of it: the weighted objective is exactly the one you wanted, and there is no correction to remember afterwards.

But do not pretend the volume is small. Negative sampling puts the impression-level positive rate at 0.128%, so negatives outnumber positives 780:1, and 90 days of impressions is:

857 M impressions/day  x  90 days  =  7.7e10 rows

Seventy-seven billion rows is a genuine compute reason to downsample — to keep only a fraction of the negatives.

The moment you downsample, an analytic correction becomes mandatory. Dropping negatives inflates every predicted probability. There is an exact formula that shifts the model’s output back down to the true rate; it is the identical mechanism as Extreme imbalance negative downsampling and the correction it forces, one order of magnitude down in scale rather than a different regime. It is also the same identity Negative sampling ran in the opposite direction for injected negatives.

Weight if you can afford the rows. Downsample and correct if you cannot. What you may not do is downsample and skip the correction, because Metrics’s notification threshold is read off an absolute calibrated probability.

8. Metrics

Offline numbers decide whether a model is allowed near production; online numbers decide whether it was any good — and here, the online number that matters cannot be read for six weeks.

8.1 Offline

Five offline measurements, each with a job. Define the four metric names first, because the table assumes them:

• Recall@25 — what fraction of the events a user actually attended would have appeared somewhere in the 25 slots. It grades the candidate generator, not the ordering.
• NDCG@25 — grades the ordering, as defined at the top of the chapter; the graded relevance scale below is what feeds it.
• ECE (expected calibration error) — the average gap between predicted probabilities and observed rates. If events scored 0.30 are attended 22% of the time, that 8-point gap is what ECE accumulates.
• Precision — the fraction of shown items that were relevant.

The graded relevance scale, and the survey hiding inside it

The graded scale is worth justifying out loud. An RSVP followed by a no-show is genuinely better than a bare click and genuinely worse than an attendance. So relevance runs 3, 2, 1, 0 rather than a plain relevant-or-not.

But look at what grade 3 requires: a rating — and only 12% of attendees leave one (Data and labels three targets and a nine day delay).

Two consequences. 88% of genuine attendances are structurally capped at grade 2. And the ideal ordering that NDCG normalizes against is itself a function of who happened to answer a survey.

So read a change in NDCG@25 only against a stable response rate. If the response rate itself moves, that invalidates the comparison; it is not a result.

One more mechanical detail. NDCG converts a relevance grade into a reward with the exponential gain 2^rel − 1, which turns 3/2/1/0 into 7/3/1/0 (Ndcg with the discount derived). So a satisfied attendance is worth seven no-shows — roughly the right ratio, given that a no-show also burns a seat the organizer could have sold.

What the offline metric cannot see

Every logged label came from an event the current ranker chose to show. So Recall@25 measured on that log is recall over the current ranker’s candidate set, which is exactly the Why offline ranking metrics disagree with online ctr trap.

The mitigation is the reserved slot 25. That is one of 25 positions, so 4% of impressions — but only half of that 4% is usable, and the reason is worth getting right:

• Half the slot-25 impressions carry a uniformly random event. Random selection is unrelated to anything you are measuring, so these are an unbiased evaluation set.
• The other half carry a cold organizer’s event (Two sided exposure and the new organizer trap). The selection rule there is “has no history,” which is exactly the variable under study. That grades nothing.

So the clean data is 2% of impressions, not 4%:

857 M impressions/day  x  2 %  =  17.2 M unbiased impressions/day

Small, precious, and the only clean data in the system.

8.2 Online, and the timing problem that makes this chapter unusual

The usual online-metric question is which number to trust. Here the question is when the number exists, because the metric you care about is not observable at all when the experiment starts.

Two abbreviations in the table. CTR is click-through rate, the share of shown items that get clicked. p90 is the 90th percentile — the value that 10% of cases exceed.

Read the “Reads out” column first. It runs from hours to six weeks, and the metric that decides the launch sits at the slow end:

How long the experiment runs

Derive the duration rather than guessing it. Note where it comes from: the calendar, not statistics.

Burn-in below is the period at the start during which behaviour is distorted simply because the feed changed and people notice novelty. The three lines add:

median lead time from recommendation to event        7 days
attendance accumulation window                      30 days
ramp + novelty burn-in                               7 days
                                                    -------
minimum experiment length                         ~44 days

Sample size is the easy half. Three terms:

• MDE — minimum detectable effect, the smallest true difference you insist on being able to see.
• sd — the standard deviation of the outcome across users.
• Over-dispersed — the counts vary more than a simple Poisson count would, which is why the standard deviation (1.10) is larger than the mean (0.42) here.

The 16 σ² / δ² formula is the standard two-arm sample size at 80% power and 5% significance:

baseline attended events / user / 30 d   =  0.42
per-user sd (over-dispersed count)       =  1.10
MDE 3 % relative                         =  0.0126

n per arm  ≈  16 sigma^2 / delta^2  =  16 x 1.21 / 0.0126^2  =  122,000 users

That 0.42 baseline does not reconcile with the volume table, and the gap is worth naming rather than papering over.

Data and labels three targets and a nine day delay puts attendances at 0.68 M/day. Divide that across the user base:

0.68 M/day x 30 days  /  40 M MAU  =  0.51 attended events per user per 30 days
0.51 / 0.42                        =  1.21,  so 21 % above the figure used here

The two numbers describe different populations. The 0.51 is a platform-wide average against a fixed MAU denominator, and it counts attendances by people who were not active across that same 30-day window. The 0.42 is the measured per-user mean over the experiment population.

Nobody has reconciled them, so use the measured one and note the direction. A smaller baseline makes a 3% relative MDE a smaller absolute delta, which needs more users — so 0.42 sizes the experiment conservatively. At 0.51 the same relative MDE needs about 83,000 users per arm instead of 122,000. Either way the conclusion holds: calendar binds, not sample size.

What the sample size implies operationally

122,000 users is nothing at 40M MAU. The binding constraint is six weeks of calendar.

That has a direct consequence: the experiment queue is the scarce resource, not the traffic. So run more arms in parallel rather than trying to make each one faster.

CUPED (controlled-experiment using pre-experiment data) removes the part of each user’s outcome that was already predictable from their behaviour before the experiment started. That shrinks the noise without touching the effect. Applied to pre-period attendance (Ab testing) it cuts variance by roughly 35% — which buys precision, and no time at all.

Randomization and interference

Randomize on the user. Then check for interference: the situation where one arm’s behaviour changes the other arm’s outcomes, breaking the assumption that the two groups are independent.

Here the mechanism is capacity. Events hold a finite number of people, so a treatment arm that fills a room has denied that room to control.

At 25 recommendations drawn from a pool of 15,000 with a 5% experiment allocation, this is negligible. At a 50/50 launch on a small metro, it is not.

How to measure it: compare a 5% arm against a 50% arm. If the effect shrinks as allocation grows, you have interference — not novelty.

9. Serving

Serving is one live feed request walked end to end, and most of the story is the single component that does the real work — the geographic index — because sizing it is pure geometry and every mistake available runs in the direction that silently makes the product smaller.

The diagram below is one feed request, top to bottom. Take the boxes in order.

Retrieval — get from 3 million events to 15,000.

1. The request arrives with a user, a position and a time.
2. Resolve location turns the raw GPS fix into two map-cell ids: a coarse res-5 cell for the retrieval ring, and a fine res-7 cell for the travel-time lookup. With no fix, it falls back to the user’s stored home cell.
3. k-ring around the res-5 cell. A k-ring is the set of all cells within k steps of a starting cell, so k=4 is 61 hexagons. That size is chosen because it guarantees coverage out to 47.6 km against the 40.2 km the product promises (Why the geo index is the whole retrieval story).
4. Each of those 61 cells owns a posting list — an index entry holding the ids of the events inside that cell, kept sorted by start time.
5. Merge the 61 lists in start-time order, apply the exact haversine test to each id as it arrives, and stop at either 21 days out or 15,000 in-radius candidates, whichever comes first.

Ranking — get from 15,000 to 25.

6. Hard filters run before anything expensive: not sold out, not cancelled, not already RSVPed by this user, not already started.
7. Feature rows are fetched, and the GBDT ranker scores all survivors.
8. Isotonic calibration per segment makes the scores read as real probabilities.
9. Diversify (MMR) applies maximal marginal relevance: repeatedly pick the next item by trading its score against how similar it is to what you already picked. Here similarity is over category and organizer, with any single organizer capped at 2 slots.
10. Position 25 is overwritten — with a uniformly random candidate on half of impressions, and a cold organizer’s event on the other half.
11. Everything shown is written to the impression log, with an exploration flag recording which of those two treatments slot 25 carried. Only the uniformly random rows are unbiased evaluation data; the cold-event rows are selected on exactly the property being studied (Two sided exposure and the new organizer trap).

flowchart TD
    REQ(["Feed request<br/>user · location · time"]) --> LOC["Resolve location<br/>GPS -> H3 res 5 (ring)<br/>+ res 7 (travel time)<br/>fallback: home cell"]
    LOC --> RING["k-ring 4 around the res-5 cell<br/>61 H3 cells<br/>guarantees 47.6 km >= 40.2 km"]
    RING --> IDX[("Geo-time index<br/>cell -> events<br/>sorted by start time")]
    IDX --> WIN["Merge, start-time ordered<br/>exact haversine <= 25 mi per id<br/>before any feature fetch"]
    WIN --> EXACT["Stop at now + 21 days<br/>or 15,000 in-radius<br/>whichever comes first"]
    EXACT --> CAND["~15,000 candidates"]
    CAND --> AVAIL{"Hard filters<br/>not sold out · not cancelled<br/>not already RSVPed<br/>not started"}
    AVAIL --> FEAT["Feature fetch<br/>user row · 15,000 event rows<br/>social join · travel-time lookup"]
    FEAT --> RANK["GBDT ranker<br/>300 trees · depth 8<br/>36M comparisons total"]
    RANK --> CAL["Isotonic calibration<br/>per segment"]
    CAL --> DIV["Diversify<br/>MMR over category + organizer<br/>cap 2 per organizer"]
    DIV --> EXP{"Exploration slot<br/>1 of 25, randomized"}
    EXP --> OUT(["25 events"])
    OUT --> LOG[("Impression log<br/>with the exploration flag")]

    style RING fill:#1d3557,color:#fff
    style EXACT fill:#1d3557,color:#fff
    style AVAIL fill:#9d0208,color:#fff
    style RANK fill:#2d6a4f,color:#fff
    style EXP fill:#bc6c25,color:#fff

The colours follow the repo key from system-design/01, read for a ranking pipeline:

• Navy — pure geometry, read out of something built offline. No model runs in either navy box.
• Green — the learned ranker. The only fitted thing on the request path.
• Red — the hard gate, authoritative at request time. It is red for the reason the key gives red: it is the step you cannot undo. A user shown a sold-out event has been given a wrong answer, not a worse one.
• Orange — the reserved slot. The one position whose contents the score does not choose.

9.1 Why the geo index is the whole retrieval story

Retrieval is one ring query on one H3 grid, so the entire design reduces to two integers: which resolution, and how many rings. Both are geometry, and both are easy to get wrong in the direction that silently shrinks the product.

Sizing the ring

The ring has to be big enough that every user standing anywhere in their own cell is guaranteed events out to the promised 25 miles. Not the average user, not the user at the centre — the worst-placed one.

That is a geometry problem with a right answer, and the two plausible-looking shortcuts both give a number that is too small.

Start from the published H3 res-5 averages, and be careful which length is which. The “edge length” the documentation gives is the hexagon’s circumradius: the distance from the centre to a vertex. For a regular hexagon the circumradius happens to equal the edge length, which is why the documentation can call it either. Every other length follows from it.

The apothem is the companion measurement — centre to the middle of an edge — and it is always the shorter of the two. Mixing them up is the first of the two errors below:

H3 res 5    edge = circumradius R                      8.54 km
            apothem = R sqrt(3)/2  (centre to edge)     7.40 km
            centre-to-centre spacing = R sqrt(3)       14.80 km
            area of a regular hexagon at that R       189.7 km^2
k-ring      3k(k+1) + 1 cells:  7, 19, 37, 61 ...

Note what is deliberately not in that block: the 252.9 km² average cell area also published for res 5.

The two published averages do not describe the same regular hexagon. An 8.54 km edge implies 189.7 km² of area; 252.9 km² implies an edge of 9.87 km. Neither is a typo — H3 cells are spherical polygons, and the two figures average different quantities over different cells.

So pick one and derive everything else from it, or the arithmetic below will not close. Ring sizing is a distance question, so keep the edge. Treat the area column in What to use instead as descriptive.

Two errors in the formula that suggests itself

The obvious formula is k × 14.80 − 7.40: walk out k cell-spacings, then back off one apothem for a user sitting at the edge of their own cell.

It is wrong twice, and both errors shrink the answer — which is the dangerous direction, because an under-sized ring raises no alarm.

Error 1: it backs off by the wrong quantity. The worst-placed user is not at the edge of their cell. They are at its far vertex, which is R = 8.54 km from the centre, not a = 7.40. Every H3 table prints the circumradius a line or two above the apothem, which is exactly how the wrong one gets subtracted.

Error 2: it uses the wrong per-ring gain. The 14.80 km figure is centre-to-centre spacing — the distance you gain heading toward a neighbouring cell’s flat face.

But the union of k rings of hexagons is not a disc. It is a notched star. A reflex notch is the inward corner where two outward-pointing cells meet, and it is the shortest way out of the shape. The guarantee is set by the shortest way out, not by the flat face.

In the notch direction each ring is worth 1.5 R = 12.82 km, not sqrt(3) R = 14.80 km.

Computing the guarantee instead

So compute it rather than trusting either formula.

The method: the distance from a point to the outside of the union is its distance to the nearest boundary segment. The guarantee is the worst such distance over every position the user could occupy inside the centre cell — so the code walks the centre cell’s own boundary and takes the minimum.

One term in the code: axial coordinates are the standard two-integer addressing scheme for a hexagonal grid, in which a cell’s six neighbours are the six offsets listed in STEPS.

The output to look at is the third line, share of 25 mi. Also read the two comment blocks — one explains why idealising H3 cells as regular hexagons is safe here, and the last one re-runs the whole thing on the other published average:

import math

SQ3 = math.sqrt(3)
R = 8.544408276          # H3 res 5 average edge = the hexagon's circumradius, km
STEPS = [(1, 0), (0, 1), (-1, 1), (-1, 0), (0, -1), (1, -1)]   # axial neighbours
NORMALS = [30, 90, 150, 210, 270, 330]                         # the edge each crosses


def _centre(i, j, R):
    s = SQ3 * R                                   # centre-to-centre spacing, 14.80 km
    return (i * s * math.cos(math.radians(30)),
            i * s * math.sin(math.radians(30)) + j * s)


def _outer_edges(k, R):
    """The segments bounding the union of every cell within k rings."""
    a = R * SQ3 / 2                               # apothem, 7.40 km
    ring = {(i, j) for i in range(-k, k + 1) for j in range(-k, k + 1)
            if max(abs(i), abs(j), abs(i + j)) <= k}
    segs = []
    for (i, j) in ring:
        cx, cy = _centre(i, j, R)
        for (di, dj), ang in zip(STEPS, NORMALS):
            if (i + di, j + dj) in ring:
                continue                          # shared with a cell we also hold
            ux, uy = math.cos(math.radians(ang)), math.sin(math.radians(ang))
            mx, my = cx + a * ux, cy + a * uy     # midpoint of that edge
            tx, ty = -uy * R / 2, ux * R / 2      # half the edge, along it
            segs.append(((mx - tx, my - ty), (mx + tx, my + ty)))
    return segs


def _dist_to_seg(p, seg):
    (x1, y1), (x2, y2) = seg
    dx, dy = x2 - x1, y2 - y1
    t = max(0.0, min(1.0, ((p[0]-x1)*dx + (p[1]-y1)*dy) / (dx*dx + dy*dy)))
    return math.hypot(p[0] - (x1 + t*dx), p[1] - (y1 + t*dy))


def guaranteed_radius(k, R=R, n=240):
    """Coverage a k-ring guarantees to ANY user inside the centre cell."""
    segs = _outer_edges(k, R)
    v = [(R*math.cos(math.radians(60*t)), R*math.sin(math.radians(60*t)))
         for t in range(7)]
    worst = float("inf")
    for e in range(6):                            # walk the centre cell's boundary
        (x1, y1), (x2, y2) = v[e], v[e + 1]
        for s in range(n + 1):
            u = (x1 + (x2-x1)*s/n, y1 + (y2-y1)*s/n)
            worst = min(worst, min(_dist_to_seg(u, g) for g in segs))
    return worst


# The regular-hexagon idealisation. It is exact AWAY FROM H3's 12 pentagons:
# a k-ring centred next to one holds 51 cells at k=4 rather than 61
# (h3.grid_disk('85080003fffffff', k) gives 6, 16, 31, 51 for k = 1..4), and
# the guarantee shrinks with it. No served metro sits on a pentagon -- they
# fall in open ocean by construction -- which is why the idealisation is safe
# here and why it has to be stated rather than assumed.
CELLS = {k: 3*k*(k + 1) + 1 for k in (1, 2, 3, 4)}
RADII = {k: guaranteed_radius(k) for k in (1, 2, 3, 4)}
print("k-ring cells      ", list(CELLS.values()))
print("guaranteed km     ", [round(RADII[k], 2) for k in (1, 2, 3, 4)])
assert list(CELLS.values()) == [7, 19, 37, 61]
assert [round(RADII[k], 2) for k in (1, 2, 3, 4)] == [8.54, 22.61, 34.18, 47.57]

# The closed form the geometry actually supports: 1.5 R per ring, offset by
# half a circumradius. A guarantee at every k, and exact on odd k.
for k, r in RADII.items():
    assert r >= (1.5*k - 0.5) * R - 1e-9
    if k % 2:
        assert abs(r - (1.5*k - 0.5) * R) < 1e-6

MILE = 1.609344
print("share of 25 mi    ",
      ["%.0f%%" % (100 * RADII[k] / (25 * MILE)) for k in (1, 2, 3, 4)])
assert RADII[3] < 25 * MILE < RADII[4]                # k=3 short, k=4 covers
assert abs(RADII[3] / (25 * MILE) - 0.849) < 1e-3     # k=3 delivers 85%

# ...on the EDGE-derived R. The area-derived one gives a different headline and
# the same decision, which is the honest way to report a figure that depends on
# which of two published averages you start from.
R_AREA = math.sqrt(252.9 / (1.5 * SQ3))               # 9.87 km, from 252.9 km^2
print("area-derived R = %.2f km -> k=3 covers %.0f%%"
      % (R_AREA, 100 * (1.5 * 3 - 0.5) * R_AREA / (25 * MILE)))
assert abs(R_AREA - 9.87) < 0.01
assert 0.97 < (1.5 * 3 - 0.5) * R_AREA / (25 * MILE) < 0.99   # 98%, still short

k-ring cells       [7, 19, 37, 61]
guaranteed km      [8.54, 22.61, 34.18, 47.57]
share of 25 mi     ['21%', '56%', '85%', '118%']
area-derived R = 9.87 km -> k=3 covers 98%

Report the 85% as a choice, not as a fact.

It is what you get from H3’s published average edge of 8.5444 km. Start from the published area of 252.9 km² instead — the disagreement flagged two blocks up — and the edge comes out at 9.87 km, which puts k=3 at 98% of the promised radius.

Both derivations are defensible, and neither reaches 100%. So the engineering conclusion is the same under both, and conservative under both: round up to k=4.

What changes is how you describe the shortfall in a design review. A number that swings from 85% to 98% on a choice made two paragraphs earlier should always be quoted with that choice attached.

                cells      guaranteed      vs a 25-mile (40.23 km) request
k = 1             7          8.54 km                 21 %
k = 2            19         22.61 km                 56 %
k = 3            37         34.18 km                 85 %   <- NOT enough
k = 4            61         47.57 km                118 %   <- covers it

k=3 does not “nearly” cover a 25-mile radius. It covers 85% of it on the edge-derived R, 98% on the area-derived one, and falls short under both.

And the shortfall is not a thin sliver. What it drops is an annulus — a ring at the outer edge — and that is where most of a disc’s area lives. The 6 km it is short of 40.23 km works out to 28% of the disc by area.

Worse, it is the outer 28%: the suburban and exurban band where the user has the fewest local options and the largest catchment. A ring that under-covers redefines the product for exactly the users the product was for, and it does so without raising a single error anywhere.

Round up to k=4.

The ring is a superset, and that is fine

A k-ring of hexagons is not a disc, so it necessarily fetches more area than the product asked for. The excess turns out to be nearly free, and the obvious way to reduce it charges you in the currency you actually care about.

61 res-5 cells cover 61 × 189.7 = 11,570 km², more than twice the 5,085 km² of the disc they have to contain. That over-read is real, and it is still the right trade.

Compare it against the obvious alternative — a finer grid needing more rings to reach the same radius:

A finer grid buys back area you were never paying much for, and charges you index seeks, which are the thing you actually pay for.

A seek is one lookup into the index to fetch one cell’s posting list. It costs a network round trip or a disk touch regardless of how few ids come back — the cost is per lookup, not per id.

Price the two surpluses against each other:

• A surplus event costs one posting-list entry and one haversine. At uniform metro density (15,000 events over 5,000 km², so 3.0 per km²), the ring holds at most 11,570 × 3.0 = 35,000 ids. Thirty-five thousand haversines is well under a millisecond.
• A surplus cell costs a seek. Going to 271 cells is four and a half times the index traffic of 61, and it buys back an over-read that was already cheap.

So the ring stays a coarse superset, and the exact predicate runs after it — on ids, before anything expensive is fetched:

61 posting lists, each sorted by start time
   -> merge in start-time order
   -> exact haversine <= 40.23 km, one float per id as it arrives
   -> stop at now + 21 days, OR at 15,000 in-radius candidates
   -> hard filters (§9.2), authoritative, at request time

The start-time sort is not decoration. It is what makes the candidate count a bound instead of a hope.

Work per request has to be capped, or the p99 belongs to whichever metro is densest. The early exit caps it in the one direction that costs nothing: truncating soonest-first drops the events furthest in the future, which are the least valuable ones anyway (Why collaborative filtering structurally fails here puts peak value at seven days out).

What that means per metro: the median metro never reaches the cap at all. A metro at three times median density holds 45,000 events in the same 21-day window, so it hits 15,000 around day 7 — which is exactly the lead time at which a recommendation is worth the most.

So: no approximate search, no index rebuild when an event’s start time changes (the posting list is re-sorted, not re-embedded), and no recall loss inside the radius up to the cap.

That last qualifier matters, and pretending otherwise would make the chapter’s own retrieval metric meaningless. In the 3x-density metro, 45,000 in-radius events exist in the window and 15,000 get scored — so 1 − 15,000/45,000 = 67% of that catalogue is never seen.

The loss is deliberate and it falls on the least valuable events. But Recall@25 counts a truncation identically to an ANN miss, because both are an attended event that was never a candidate. That is why Offline measures recall at all. If “no recall loss inside the radius” were literally true, Recall@25 would be pinned at 1.0 by construction and would be a metric with no job.

Compare against the alternative honestly

Declining to use an approximate index is only a strong answer if you can price the index you declined — and the arithmetic is worse than most people expect.

The alternative is a single global HNSW graph over 3M event vectors, with geography applied as a filter, at the 0.50% selectivity derived in Model choice. There are two ways to apply that filter, and both fail.

Pre-filtering enforces the filter during the graph walk. It degenerates because the graph was built over all 3M nodes, so the search keeps stepping onto ineligible neighbours and stalls (When it is not fine say these unprompted).

Post-filtering retrieves first, then discards whatever is out of range. It fails arithmetically, and the arithmetic is short.

Two terms for the block below. E[survivors] is the expected number of candidates left after filtering. The count of survivors follows a Poisson distribution — the standard distribution for “how many of many independent rare things happened” — whose mean is conventionally written λ:

retrieve top 500 by embedding, then apply the geo filter
E[survivors]      =  500 x 0.0050  =  2.5
P(zero survivors) =  e^-2.5        =  8.2 %

One request in twelve returns an empty feed. And it is not a random twelfth — it is the users in thin metros, every time.

Note how the second line is computed. 1 − E[survivors] is not a probability; that shortcut is only harmless when λ ≪ 1, and at λ = 2.5 it would have returned a negative number. The Poisson zero-probability e^-λ is the right form.

Then price the fix. To fill 25 slots you need 25 / 0.005 = 5,000 retrieved in expectation. To fill them 99 times out of 100 you need 7,609, because 7,609 × 0.005 = 38 expected survivors is where the Poisson tail clears 25 with 99% probability.

Line the two designs up:

HNSW + post-filter:  traverse 7,609 nodes of a 3M-node graph
                     ->  38 in-radius events, approximate recall
this design:         61 index seeks
                     ->  all 15,000 in-radius events, exact

The geo constraint is not a filter to be applied to a similarity search. It is the partition key — the field you split the data by in the first place, so a query touches only the shard it needs instead of searching everything and throwing most of it away.

9.2 The staleness problem, and why capacity is not a feature

Some facts change faster than any feature pipeline can refresh them, and a line has to be drawn between what may live in the feature store and what must be read live at request time.

Fill rate changes by the minute near a popular event. A feature store is the database that holds precomputed feature values for serving, refreshed on a schedule. If capacity lives in the ranker’s feature store with a 15-minute refresh, you will recommend sold-out events for 15 minutes at a time — and the p99 of that window lands exactly on the events that sell out fastest, which are the ones the ranker loves most.

Sold-out is a hard filter read from the authoritative counter at request time, not a feature read from a warehouse. Authoritative means the counter that the booking path itself writes to, so it cannot be behind.

The cost of doing it live is small. One lookup per candidate against Redis — an in-memory key-value store — all batched together. Fetching 15,000 keys in a single pipelined MGET (the command that fetches many keys in one round trip) costs about 6 ms.

Note when it runs: after the exact haversine cut in Why the geo index is the whole retrieval story. That ordering is why the geo predicate is applied to ids rather than to fetched rows.

Fill rate and sold-out are different things, and only one of them is a feature. Fill rate stays a feature, because 0.6 is genuinely informative. Sold out is a boolean gate.

The same argument applies to cancellations and to “the event already started.”

9.3 Rollout, and the one thing a canary here cannot check

Shipping a change here has a problem no other chapter in this track has: the quality signal does not exist yet when you have to decide whether to proceed.

The standard sequence is shadow (run the new system alongside the old on real traffic, compare, serve nothing), then canary (serve a small slice of real users), then ramp (increase that slice). All three apply here, with a wrinkle this chapter’s own physics creates. The decision metric reads out in five to six weeks (Online and the timing problem that makes this chapter unusual), so a canary that runs for a day has access to exactly one fast signal: RSVP rate, which The rsvp to attendance funnel is not uniform and that is the trap proves can move in the opposite direction to the thing you sell.

So the canary gates on safety, not on quality. p99 latency, empty-feed rate, click-to-sold-out rate (The sold out recommendation), and the served distance distribution — served, not attended, because attendance is thirty days away. Quality waits for the six-week experiment and there is no shortcut; a canary that green-lights on RSVP lift is the The rsvp to attendance funnel is not uniform and that is the trap trap wearing a deployment badge. Canary on one metro rather than a traffic slice, which also bounds the finite-capacity interference Online and the timing problem that makes this chapter unusual flags.

And shadow the geo index separately from the ranker, by replaying a day of requests through the new ring and diffing candidate sets against the old one. That is the check the rest of the system cannot perform: a resolution mistake of the kind Why the geo index is the whole retrieval story dissects returns a seventh of the candidates it should, and every downstream metric — CTR, NDCG, latency, error rate — either improves or stays flat when that happens. Candidate-set size is the only place an under-fetch is visible, and nothing else in the pipeline is watching it.

10. Scale and cost

The design now turns into a latency budget and an annual bill. Both are worth doing in an interview because they end somewhere surprising: the latency is dominated by fetching features rather than by the model, and the money is dominated by storing users rather than by anything to do with events.

40 M MAU · 2 sessions/week · 3 feed loads      =  240 M requests/week
                                               =  397 QPS average
                                               ~ 1,200 QPS peak

Now the per-request budget, at the 15,000 candidates Why the geo index is the whole retrieval story derives.

p50 is the median request — half of requests are faster. p99 is the deadline Framing set. Both are quoted, because a budget stated only at the median hides exactly the requests the deadline was written for.

Two terms in the feature-fetch line. A columnar gather pulls one field at a time across many rows out of a column-oriented layout, which is far faster than assembling 15,000 whole records. The arena it reads from is a large block of memory the serving process owns outright, so no network or disk is involved.

The two lines to watch are feature fetch: 15,000 event rows at 25 ms and GBDT scoring at 36 ms — the model is not the biggest line, and the feature work around it adds up to more:

resolve location -> res 5 ring cell + res 7 travel cell   1 ms
geo index: 61 posting-list seeks + start-time merge      22 ms
exact haversine + dedupe over the merged ids              3 ms
hard filters: 15,000-key MGET, pipelined                  6 ms   (network)
feature fetch: 1 user row                                 3 ms   (network)
feature fetch: 15,000 event rows                         25 ms
   (34 MB columnar gather from a process-resident
    arena; 2.3 KB/row x 15,000, ~6.9 GB across
    all 200 metros, so every metro stays hot)
social join: friends x candidate RSVP lists              20 ms
travel-time lookup: 15,000 array reads                    2 ms
GBDT scoring: 36 M comparisons                           36 ms
calibration + MMR + exploration                           8 ms
response: hydrate 25 events, serialize, write            12 ms   (4 network)
                                                       -------
                                                        138 ms  p50
                                                        220 ms  p99

The response line is small only because the funnel already ran. Hydration means fetching the human-facing payload — title, image URL, venue, price — and here it happens for 25 events rather than 15,000, because it happens after ranking rather than before. The place a budget like this usually breaks is hydrating before ranking rather than after — sibling chapter Scale and cost spends 105 ms on payload and serialization for the same shape of product because its payload is richer, and that is the line to check first when a feed misses p99.

The p50-to-p99 spread is 1.6x rather than the 2.5x you would expect, and the reason is the Why the geo index is the whole retrieval story cap.

Once the candidate list is bounded at 15,000 by the start-time early exit, work per request is fixed. The only variance left is the three network hops plus garbage-collection pauses.

Without the cap, the slowest 1% of requests would simply be whichever metro is densest. A metro at three times median density would spend 3 × 36 = 108 ms on GBDT scoring alone — over budget before anything else has run.

So 220 ms against a 250 ms deadline is a real budget with real headroom. But the headroom is 30 ms, not the comfortable 168 ms a design assuming a few thousand candidates would have predicted.

Fleet size

Size the fleet on the 125 ms that is actually CPU. The 13 ms of network waits do not occupy a core:

1,200 QPS peak x 125 ms CPU  =  150 cores busy
x 3 for p99 headroom and failover =  ~450 cores
at 30 usable cores per server     =  ~15 servers

The 385 GB feature-store line is the only one whose punchline depends on a number, so derive it. Watch the relative sizes of the four rows — the events line is the small one:

40 M users x (1 KB profile + 500 seen-event ids x 8 B)     200 GB
social adjacency, 40 M x ~150 friends x 8 B                 48 GB
3 M live event rows x 2.3 KB                                 6.9 GB
event -> RSVP attendee lists, 3 M x 34 x 8 B                 0.8 GB
                                                          --------
                                                           256 GB
x 1.5 replication and fragmentation                       ~385 GB

The store is sized by people and their friends; events are 7 GB of 256. That is Why collaborative filtering structurally fails here restated in gigabytes — the item carries almost nothing, so almost nothing is stored about it, and the two structures that dominate are the user’s own history and the social graph that Why collaborative filtering structurally fails here identified as the one surviving collaborative signal.

The whole system costs less than two engineers, and the expensive part is the feature store, not the model. That is the normal shape for a tree-based recommender and it is worth stating out loud, because candidates routinely propose an architecture whose serving cost exceeds the revenue of the surface it powers. For contrast, see Scale and cost where the same reasoning runs at 1,000x the QPS.

11. Failure modes

Seven ways this system breaks in production, each traced from a symptom to the component that caused it, and each with a detector and a control. Two of them — the sold-out recommendation and the new-organizer trap — are the ones an interviewer is most likely to raise unprompted.

11.1 The sold-out recommendation

The first failure is the one where the model’s best feature and the bug’s root cause turn out to be the same number.

14:02  event E: capacity 120, RSVPs 118, fill_rate 0.983
       -> ranker LOVES fill_rate 0.98 (strong popularity signal)
       -> E ranked #1 for 340 users in this metro
14:07  RSVPs hit 120. SOLD OUT.
14:07  feature store snapshot is from 13:55. fill_rate still reads 0.983
14:07 - 14:10   E served as the #1 recommendation to 61 more users
       -> 61 clicks -> 61 "Sold out" pages
       -> 9 of them leave the app within 30 s

The ranker’s favorite feature and the failure’s root cause are the same number. Fill rate is the strongest single content signal in the model and it is monotonically approaching the value at which the item becomes unshowable.

Detection: click-to-sold-out-page rate, alerted at > 0.5%. Control: the The staleness problem and why capacity is not a feature request-time gate. Second-order control: damp fill_rate above 0.9 so the ranker stops preferring almost-full events over half-full ones — the incremental signal above 0.9 is mostly “about to be unavailable.”

11.2 The 40-mile drive

The second is a recommendation that is correct about interest and wrong about follow-through, and it lands hardest on exactly the users with the least data.

user: sparse profile, 2 lifetime RSVPs, both within 3 miles
event: a well-rated festival 38 miles away, huge social signal (4 friends going)

model score 0.71 -> ranked #2
user RSVPs. Does not attend.

what the model knew:  distance band = "40+", P(attend|RSVP) at that band = 0.44
what the model optimized: RSVP, because the attendance head had 2 observations
                          for this user and shrank to the metro prior

The distance decay is in the model. The problem is that the user-specific willingness to travel was estimated from 2 observations and shrank to a metro mean that includes people with cars and free Saturdays.

Detection: p90 distance of attended events, split by user history depth. Control: make the shrinkage asymmetric — shrink toward the conservative end for sparse users, because the cost of an over-long recommendation (a no-show, an organizer’s empty seat, a user who stops trusting the feed) exceeds the cost of an under-ambitious one. This is a cost-matrix argument, not a modelling one (Choosing a threshold from the cost matrix).

11.3 The sparsity of any individual user

The third is less a failure than a permanent condition, and being honest about its size is what makes the rest of the design defensible.

The percentiles below read as: the median user (p50) has 3 lifetime RSVPs, the 90th-percentile user has 14, and even the 99th-percentile user has only 61.

lifetime RSVPs per user:   p50 = 3    p90 = 14    p99 = 61
users with 0 RSVPs:        41 % of MAU

For 41% of the audience there is no user history at all, and for the median user there are three data points. Everything the model knows about the median user is inherited from the aggregate.

Consequences, in order of severity:

1. The feed collapses to popularity for most users. Which is not a bug — popularity is the correct prediction under no information — but it produces a homogeneous feed that generates no new signal.
2. The feedback loop closes. Popular events get shown, get RSVPs, get more popular. Detecting drift without labels the real production problem.
3. Cold-start users churn before they generate data.

Control: an explicit onboarding preference capture (category and neighborhood picks) is worth more than any modelling change, because it converts 0 observations into ~5 informative ones at zero latency cost. Measured: 30-day attendance for onboarded users 0.61 vs 0.29 for non-onboarded (onboarding is self-selected, so this is a correlation and not the effect of onboarding; the causal read needs a holdout, and the honest expectation is that the true effect is smaller than 2.1x).

11.4 Seasonality, which looks like model rot

The fourth is a slow decline that looks like the model going stale and is actually the calendar moving underneath it.

model trained on Nov 1 - Jan 15, evaluated by month:

month        NDCG@25     top category by impression share
Feb            0.309     holiday markets  (18 %)   <- category no longer exists
Mar            0.281     holiday markets  (14 %)
Apr            0.244     holiday markets  (11 %)

The model learned a December expectation for what categories people want and carried it into spring. Two different kinds of change are happening at once here: covariate shift on the item side, meaning the mix of things being scored has changed while the underlying preferences have not, and concept drift on the user side, meaning the relationship between features and outcome has itself changed (Drift three different failures with three different signals).

Detection: PSI, the population stability index — a single number summarizing how far one distribution has moved from another — computed weekly between the category mix of what was impressed and the category mix of what is actually live. A rising PSI with a stable catalogue means the catalogue moved on and the model did not. Control: the hourly p_rsvp retrain (Training) handles most of it; add explicit week_of_year and days_to_nearest_holiday features so the model has a place to put seasonality rather than smearing it into category priors.

11.5 Recurring events — the one case where you have history and throw it away

The fifth is not a modelling failure at all. It is a database schema quietly destroying the only usable item history in the system.

A weekly run club creates a new event object every Monday. Treated as 52 independent items, each has 9 RSVPs and is permanently cold. Treated as one series, it has 470.

This is the only place in the chapter where a genuine item embedding is learnable, and the naive schema destroys it. Join on series_id (organizer + title similarity + recurrence rule), and inherit attendance rate, rating, and no-show rate from the series. Roughly 22% of events belong to a series, and they account for 34% of attendance — so this is a third of the problem, recovered by a schema decision.

There is a trap on the other side of that fix: inherited signal must decay. A series that was excellent for two years and changed venue last month should not carry two years of ratings at full weight. Weight past instances by recency with a half-life of about 8 instances — meaning an instance 8 occurrences ago counts half as much as the most recent one, one 16 ago counts a quarter, and so on.

11.6 Two-sided exposure and the new-organizer trap

The sixth is a feedback loop on the supply side of the marketplace, and the fix is a single reserved slot whose cost you should be able to state to two decimal places.

The loop closes on itself:

new organizer -> no attendance history -> weak organizer features
              -> low rank -> no impressions -> no attendance history

Nothing breaks that cycle from inside, so the platform’s supply dies quietly.

Slot 25 carries two rotating treatments, and they are not the same experiment.

• On 50% of impressions it is filled uniformly at random from the candidate set. That is the unbiased evaluation log.
• On the other 50% it is filled with a cold organizer’s event. That is the supply-ramp intervention.

Separating them is not pedantry. A slot reserved for cold events is biased by construction, because its selection rule is “has no history” — exactly the variable under study. So only the uniform half is usable as unbiased evaluation, and only the cold half ramps supply.

The cost is the same either way, because both treatments replace a ranked event with one expected to convert at 0.40x. Price the intervention rather than arguing about it:

share of module RSVPs coming from slot 25          2.1 %
expected RSVP rate of the slot-25 treatment
  vs the ranked event it replaces                  0.40x

cost of reserving slot 25, either treatment
   =  2.1 % x (1 - 0.40)  =  1.26 % of module RSVPs
module drives 11 % of platform RSVPs
   ->  1.26 % x 11 %      =  0.14 % of platform RSVPs

benefit, cold half:  median time-to-first-attendee for a new organizer
                     38 days  ->  12 days   (measured at the full slot; at
                     half the impressions the ramp is correspondingly slower,
                     which is the price of the split)
benefit, random half: 34.3 M requests x 0.5 slot
                     =  17.2 M unbiased impressions/day

0.14% of RSVPs buys a 3x faster supply ramp on one half and the only unbiased evaluation data in the system on the other (Offline). Two payoffs from one slot, at half the volume each — and if you ever need more statistical power on the eval side, the dial is the split, not the slot count. Frame it that way and the trade stops being a values argument.

11.7 Timezones, and the bug that ships every year

The seventh is the cheapest to prevent and the most reliably shipped anyway, because it only manifests on two weekends a year.

Here is the bug in one line. 2025-11-02 01:30 America/New_York occurs twice that night: clocks go back, so the same local time happens, then happens again an hour later.

So a local time stored without its zone is genuinely ambiguous — not hard to resolve, ambiguous — on one weekend a year in every country that observes DST, daylight saving time.

Store the UTC instant and the venue’s IANA zone id separately. Never a bare local time.

• UTC is the single global reference timeline that never shifts.
• An IANA zone id is the standard identifier for a place’s timezone rules, such as America/New_York. It is what converts a UTC instant back into what a clock on the venue wall said.

Then split the usage. Every “is it over / soon / this weekend” predicate runs on the UTC instant. Only display and the hour_of_week feature use venue-local time.

And note that last word: venue-local. An event happens where it happens, so rendering it in a travelling user’s device timezone is also a bug.

11.8 Summary

All nine failures in one table, including two — zombie recommendations and RSVP-optimized ranking — that were derived earlier in the chapter rather than in this section.

12. Alternatives considered and rejected

Thirteen designs a reasonable engineer would propose, each with the honest reason it appeals and the specific number that rules it out.

As you read the right-hand column, count how many rows point back at Why collaborative filtering structurally fails here. The item having no history is not one constraint among many. It is the constraint that keeps reappearing.

Two names in the table need defining. item2vec trains word-embedding machinery on sequences of items a user interacted with, so items appearing in similar contexts get similar vectors. DCN is Deep & Cross Network, a relative of wide & deep built to learn feature combinations automatically.

13. Interviewer pushback

Nine questions an interviewer actually asks on this problem, with what each one is testing and the answer as you would say it out loud. The third is the chapter’s trap: a genuinely good number that means the opposite of what it looks like.

“Why not just use collaborative filtering? It works everywhere else.” Testing: whether cold start is a word you know or a constraint you can quantify. Because the item’s information and the item’s value are anti-correlated in time here. A 64-dim item vector needs on the order of 640 observations to be about the item rather than about the regularizer, and the median event has 9 RSVPs at seven days out — which is exactly when a recommendation is worth the most. Only 0.3% of events ever reach 640, and they reach it after they have already happened.

So for 99.7% of the catalog, matrix factorization returns the prior mean. I would have built a popularity ranker carrying 64 wasted floats per row.

What I do instead is push the collaborative machinery onto entities that persist: the organizer, the series, the venue, the category. An event has 9 interactions; the organizer who runs it monthly has 400.

“Why is raw latitude and longitude a bad feature?” Testing: whether you understand what a tree can and cannot express. Because the target depends on the difference between two positions, and a split depends on one absolute position. “Within five miles of the user” is a disc, and a tree can only build axis-aligned staircases: four inscribed bands cover 55% of a disc, eight cover 79%, sixteen cover 90%. And a band is not a leaf — it is one y-split plus two x-splits — so sixteen bands is about forty-eight leaves.

Worse, the disc moves. A user in Oakland needs different leaves than one in San Jose, so the leaf budget scales with the number of distinct user neighborhoods and the feature is effectively unlearnable. Precompute the haversine and “within five miles” becomes one split.

I would still keep H3 cells as categoricals, because “this neighborhood is desirable” is a genuine absolute effect that distance cannot express. But the invariant quantity has to be computed before the model sees it.

“Your RSVP rate is up 9%. Ship it?” Testing: whether you take a good number at face value. It is the trap in this chapter. Not on that number. RSVP is free and attendance is not, so the two diverge in a predictable direction: P(attend given RSVP) is 0.88 for paid events and 0.41 for free ones, 0.71 under five miles and 0.44 past fifteen. An RSVP-optimized model finds exactly the events people flake on.

Here is the arithmetic. The overall 0.62 implies the current mix is 55% free, since 0.41f + 0.88(1−f) = 0.62. If an RSVP-optimized model moves that mix to 72/28, attendance per RSVP falls from 0.62 to 0.542 — and 9% more RSVPs at the lower rate is 4.8% fewer attendances. I would derive that baseline rather than assume 50/50: assuming it gives 0.645 and reports the damage as 8.5%, which is nearly double and would not survive review.

So I would look at attended events per user over 30 days, plus the no-show rate as a guardrail. The structural fix is to train two heads — a fast marginal on RSVP and a slow conditional on attendance given RSVP — and multiply them, because the conditional stays stable even though its labels are nine days old.

“How long does the experiment run?” Testing: whether you have thought about when the label exists. About six weeks, and the constraint is calendar, not traffic. Median lead time from recommendation to event is 7 days, I want a 30-day attendance window, and I want a week of novelty burn-in. That is 44 days minimum.

Sample size is trivially met. The baseline is 0.42 attended events per user per 30 days with a standard deviation of 1.1, so a 3% MDE needs about 122,000 users per arm — against 40M MAU. CUPED on pre-period attendance cuts variance about 35%, which buys precision but not a single day.

The operational implication is that the experiment queue is the scarce resource, so I would run many arms in parallel rather than trying to make each one faster.

“Walk me through candidate generation. Which ANN index?” Testing: whether you reach for a tool or for the constraint. None, and the reason is not the selectivity number people expect. A 25-mile disc is 5,085 km² and a metro is about 5,000, so the candidate set is a metro’s live catalog: 3M over 200 metros is 15,000, which is 0.50% selectivity. That is below the ~1% where filtered HNSW degenerates, but only by 2x, so I would not hang the decision on it. The decision is that geo is an exact predicate — showing someone an event 40 miles outside the radius is a wrong answer, not a worse one — and an approximate index cannot promise exactness.

I would price the alternative anyway. Post-filter 500 retrieved and you expect 500 × 0.005 = 2.5 survivors, which by Poisson is e^-2.5 = 8.2% empty feeds, and it is always the same thin metros. To fill 25 slots 99 times in 100 you would traverse 7,609 nodes of a 3M-node graph to yield 38 in-radius events — where 61 index seeks yield all 15,000 of them exactly.

So: partition. H3 res 5, k-ring 4, 61 cells. And I would size that ring from the geometry rather than from the obvious formula. The guarantee is (1.5k − 0.5) · R, with R the 8.54 km circumradius. Two reasons the obvious formula is wrong: the binding direction is the notch between two outward cells, where a ring is worth 1.5R = 12.8 km rather than the 14.8 km centre spacing, and the worst-placed user sits at their own cell’s vertex, not its edge. That gives 34.2 km at k=3 and 47.6 km at k=4 against the 40.2 km required — so k=3 delivers 85% of the advertised radius and k=4 is the answer.

Each cell holds a posting list sorted by start time. I merge in that order, stop at 21 days or 15,000, and run the exact haversine on ids before any feature fetch. 15,000 is small enough to score every one, so there is no recall loss inside the radius up to that cap. And I would say where the cap bites rather than claim it does not: a metro at three times median density holds 45,000 in-radius events in the 21-day window, so 67% of its catalogue is truncated away, soonest-first — and Recall@25 counts that identically to an ANN miss.

“You recommended a sold-out event. How did that happen and how do you stop it?” Testing: whether you can trace a failure to a specific component. Fill rate is the ranker’s favorite content feature, and it is also the countdown to unavailability. The model most loves an event at 0.98 full, which is minutes from being unshowable. If capacity comes from a feature store with a 15-minute refresh, there is a 15-minute window in which sold-out events rank first — and it lands preferentially on the events that sell fastest.

Two fixes. First, sold-out becomes a hard gate read from the authoritative counter at request time; 15,000 keys in one pipelined MGET is about 6 ms. Fill rate itself stays a feature, because 0.6 is genuinely informative. Second, I would damp fill rate above 0.9, since the incremental signal up there is mostly “about to disappear” rather than “popular.”

“The median user has three lifetime RSVPs. What can you actually personalize?” Testing: intellectual honesty about the data you have. Very little, and I would rather say the number than imply otherwise. Shrinking a per-user day-of-week histogram toward the metro prior with a pseudocount of 20 gives the user’s own three observations 13% of the weight — 3 / (3 + 20). And 41% of monthly actives have zero RSVPs at all.

So for most of the audience the model is predicting the city, not the person. The honest framing is “learn the city well, then nudge.”

That is why the highest-return investment is not a better model but an onboarding step that captures category and neighborhood preferences. It converts zero observations into about five informative ones at no latency cost, and 30-day attendance for onboarded users is 0.61 against 0.29. I would flag that as a correlation, though: onboarding is optional and therefore self-selected, so the causal number needs a holdout arm.

“NDCG went from 0.317 to 0.412. Ship it?” Testing: whether you check how the number was produced. That gap is the signature of a random split, not a better model. fill_rate and rsvp_velocity_6h are features, so a random split puts impressions of the same event on both sides and lets a validation row be scored using knowledge of how popular that event became. When I re-split by time, the same model reads 0.317 — and in the one case where we shipped the leaky-split winner it lost 1.8% on RSVP online while the temporal-split model gained 2.4%. So: temporal split, boundary placed before any validation event was created, and the leaky number is not a tiebreaker.

“You have one engineer and one quarter. What ships?” Testing: whether you can sequence under a real constraint. The geo-time index, six distance buckets, the 168-way hour-of-week bucket, the social-graph join, and a LightGBM ranker over those — plus the sold-out gate and a temporal split. That is most of the achievable quality, because distance, time, and friends are the three features carrying the signal, and none of them requires an embedding or a GPU.

What I would defer: the travel-time matrices, the content tower, the two-head attendance decomposition.

What I would not defer, despite the temptation, is the reserved slot 25. It costs 0.14% of RSVPs, and its uniformly random half is the only source of unbiased evaluation data I will ever have. I would also log which of its two treatments each impression carried, because the cold-organizer half is selected on the variable under study and grades nothing. A quarter from now I will want six weeks of that log already sitting there.

14. Cheat sheet

Fourteen questions and the one-line answer to each, for the last five minutes before an interview. Every line here is derived somewhere above; none of it is worth memorizing without the derivation behind it.

Next: 08 — Ad Click Prediction — where the item has plenty of history, the QPS is a thousand times higher, and the predicted probability gets multiplied by a bid, which is what makes calibration stop being optional.