How to design an event recommender

Concerts, meetups, classes, markets: the job is to recommend the local events a user might actually want to attend.

In this lesson, we’ll build that recommender end to end, starting from the one fact that breaks the usual playbook. Local events are items with no interaction history, and they never accumulate one that is useful in time. An event exists once, at one place and time, and is worthless the moment it is over. By the end you’ll be able to reason about any problem where the item you recommend expires: how to route the collaborative signal to something that persists, how to split a delayed label into a fast part and a slow part, and why the geometry beats an approximate index here.

Input and output. In: one user (current location, past actions, friend list, current time) and the live catalog of every event happening anywhere. Out: 25 ranked events for the home feed plus a smaller “near you this weekend” module, produced in under 250 ms.

RSVP is the button a user presses to say they intend to go. It is free and instant, and it is not the same as showing up. That gap is the central problem of the lesson.

Two ideas from elsewhere on this site are restated where they are needed below: how ranking quality is scored (ranking and recommendation metrics) and why gradient-boosted trees still beat neural nets on tabular data (the ensembles chapter).

The core problem: cold start that never ends

Most recommenders assume an item you can learn about. A movie accumulates ratings for a decade; a product accumulates purchases indefinitely. An event has no interaction history before it happens and no value after.

Cold start is the standard name for having to recommend an item with no data on it. In most systems it is an edge case that decays: wait a week and the item warms up. Here it does not decay, because the item is destroyed before it warms up. In this system cold start is the permanent, normal case rather than an exception.

That single fact disqualifies the default architecture. Collaborative signal cannot carry the ranking, because the interaction history an item accumulates peaks at exactly the moment its value hits zero. Content and context have to carry it instead, and the two features that carry the most are where the user is and when they are free.

The system at a glance

This table lists every learned or fitted component, so that when a later section names “the attendance head” its inputs and outputs are already defined. Several rows say “Not a model” on purpose: many of the good decisions here are decisions not to learn something.

Four terms the table uses:

• Offline: runs on a schedule; its output is stored.
• Online: runs inside the 250 ms of a live feed request.
• NDCG@25: normalized discounted cumulative gain over 25 slots. It scores an ordering: reward each relevant item, discount it by how far down the list it sits, sum, then divide by the best score any ordering of the same items could reach. 1.0 is perfect.
• Calibrated: a probability that means what it says. Events scored 0.30 are attended about 30% of the time, which lets you compare the number against a threshold, not only against other scores.

flowchart TD
    subgraph OFF["Built offline, on a schedule"]
        CT["Content tower<br/>text + image → 64-dim vector<br/>once per event at creation"]
        GI["Geo-time index<br/>map cell → events, sorted by start time"]
        RF["Shrunken rate features<br/>user / metro×category×hour / global"]
        H1["p_rsvp head (GBDT)<br/>label: RSVP, same-day<br/>retrain hourly"]
        H2["p_attend_given_rsvp head (GBDT)<br/>label: attendance, 9-day delay<br/>retrain weekly"]
    end
    subgraph ON["Per request, under 250 ms"]
        R["Retrieve: geo + time hard filter<br/>3M events → ~15,000 candidates"]
        S["Score = p_rsvp × p_attend_given_rsvp"]
        C["Calibrate (isotonic, per segment)"]
        D["Diversify + reserved slot 25"]
        F["25 events"]
    end
    GI --> R
    CT --> S
    RF --> S
    H1 --> S
    H2 --> S
    R --> S --> C --> D --> F

The “where its labels come from” column is the whole shape of the problem. Two components have no labels, one has labels that arrive instantly and mean the wrong thing (RSVP), and one has labels that mean the right thing (attendance) and arrive nine days late. No component has labels from the item’s own history, because that history does not exist when you need it.

Scale and constraints

A few clarifying answers shape everything downstream:

• Success is attendance, not RSVP, and it is observed 7–17 days after the recommendation.
• The catalog is local. Geo is a hard constraint, not a soft preference, which rules out approximate search (below).
• A typical user attends 0.42 events per 30 days and has 3 events of lifetime history. The label is rare and per-user data is almost nonexistent.
• There are organizers on the supply side, which makes this a two-sided market with an exposure-fairness problem.

Why collaborative filtering fails here

Collaborative filtering recommends to you what people who behaved like you also liked. It learns purely from the table of who interacted with what and understands nothing about the items themselves. Its standard form is matrix factorization: represent each user and each item by a short vector of d numbers (the latent dimension, typically 64), and predict how much a user likes an item by multiplying their two vectors. The numbers are fitted so the products reproduce the observed interactions.

The counting argument. An item’s vector holds d free numbers, and each observed interaction supplies roughly one number’s worth of evidence. As a rule of thumb you need about 10 · d ≈ 640 interactions before the fitted vector describes the item instead of the fitting penalty. With fewer observations than free parameters, the fit is underdetermined: infinitely many vectors explain the data equally well, and which one you land on is decided by the regularizer, the penalty that pulls unconstrained numbers toward a default (usually average behavior).

Now measure the event at the moment a recommendation is worth making. The median event gets 34 RSVPs over its whole life but only about 9 by seven days out, which is when a recommendation has peak value. So you are solving for 64 numbers with 9 facts, underdetermined by 7x. (Those 34 lifetime RSVPs reconcile with the platform’s 1.1M RSVPs/day only through a mean event lifetime of ~93 days, not the median of 17: the distribution is heavily right-skewed by festivals and conferences listed months ahead. Confusing that skewed median for the mean throws a creation-rate estimate off by 5.5x.)

What an underdetermined fit returns is not noise. It returns the regularizer’s prior mean, which is the global popularity term with no item identity in it at all. So you have not built personalization; you have built a popularity ranker carrying 64 wasted numbers per row. Only about 0.3% of the catalog ever reaches 640 lifetime RSVPs, and those reach it only after the event has already happened.

Information and value move in opposite directions

Follow the median event from creation to the day after it happens:

For movies and products the two move together: the longer an item lives, the better you know it and the more people it can still serve. For events they trade against each other.

The one collaborative signal that survives

One collaborative signal does survive: the user’s social graph crossed with the event’s current RSVP list. It survives because it is not a property of the event’s history; it is a property of who has already said yes, which exists from the very first RSVP.

The numbers are lift, the factor by which a user’s chance of going rises relative to their own baseline:

0 friends attending      1.0x   (baseline)
1 friend  attending      4.7x
2 friends attending      7.9x
3+ friends attending    11.2x

The strongest single feature in the system is a join between two tables that both exist on day one. It is collaborative filtering that does not need the item to have a history, because the “collaboration” is over people already known.

The ML objective

The model does pointwise binary classification: it looks at one (user, event) pair in isolation and predicts a probability, and the feed is sorted by that probability.

p = P(user u attends event e | nothing else on u's calendar clashes, u sees e)

Three design decisions hide in that line:

1. The target is attendance, not RSVP. They are different events with different rates (below), and optimizing the wrong one looks like a win on the dashboard while losing what the business sells.

2. | u sees e is not free. An impression is one event appearing in one user’s feed. Training data is only logged impressions from the current ranker, so the model never sees anything that ranker refuses to show. This is the standard feedback loop; the mitigation is an exploration slot, not a clever loss.

3. Pointwise, not pairwise or listwise. A pairwise loss trains on “is A better than B”; a listwise loss trains on whole orderings. Both learn an order and throw the scale away: a model that scores everything 0.9 and one that scores everything 0.02 are identical to them. Here you need the scale, for two reasons.

• The feed has a quality floor. Showing 25 events when only 4 are any good is worse than showing 4. Deciding that requires knowing whether 0.9 or 0.02 is the real number.
• The notification path fires on an absolute threshold, set by weighing the cost of interrupting someone against the value of them going, which only works if the score is a real probability.

So the recipe is: fit a pointwise logistic loss, rank by the calibrated score, then threshold that score against a cost matrix.

Data and labels: three targets and a nine-day delay

Four outcomes are observable, in increasing order of value and decreasing order of availability. A geofence ping is a signal from the phone that the user physically entered a circle around the venue, which is how attendance gets observed for events with no ticket or door scan.

The complete label lands nine days after the recommendation: the recommendation is served ~7 days before the event, the event happens, and the rating survey closes two days later. No pipeline engineering fixes this. The event genuinely has not happened yet. Three consequences follow:

1. You cannot gate a daily A/B on the real metric, because the readout lags exposure by 9 days minimum (30+ days for an attendance-per-user window).
2. Seasonal drift is discovered late: a shift that begins on a Friday is not visible in complete labels until nine days later.
3. You will be tempted to train on RSVP because it is available now. Do not, at least not naively.

The RSVP-to-attendance funnel is not uniform

Overall, 62% of RSVPs become attendance. That single number is a trap. Split it and it falls apart:

An RSVP has no price; a ticket does, and so does a long drive. Free and distant events are systematically over-RSVPed and under-attended, so a model trained on RSVP is trained to find exactly the events people will flake on.

You can price the damage. The overall 0.62 is a weighted average of the free (0.41) and paid (0.88) rates, which pins the current mix at 55% free, 45% paid. A model trained on RSVPs favors free events and shifts that mix, say to 72/28, dropping attendance-per-RSVP to 0.54. Combined: it wins on its own objective (RSVPs up 9%) but each RSVP is worth less, so attendance falls 4.8%. (Deriving the 55% from the data instead of assuming an even 50/50 mix matters: 50/50 would overstate the damage as 8.5%, inflated by an assumption you made up.)

The fix: two heads, one trained late

Train one model with two outputs and multiply them:

score = p_rsvp(u, e) · p_attend_given_rsvp(u, e)

The marginal p_rsvp is the overall RSVP rate. It retrains hourly on same-day labels, tracking the fast-moving catalog. The conditional p_attend_given_rsvp is the follow-through rate among people who already said yes. It retrains weekly on labels that are nine days old, and that staleness is fine: the mix of events changes hourly, but the probability that a person who RSVPed to a free event 20 miles away actually shows up does not change from week to week. Splitting a delayed label into a fast marginal and a slow conditional means only the fast part needs fresh data.

Concretely, a paid event 2 miles away with 2 friends going (p_rsvp 0.20, p_attend_given_rsvp 0.62) scores 0.124, while a free event 20 miles away and alone has the higher RSVP probability (0.26) but lower follow-through (0.44), scoring 0.114. It ranks below, which is the whole point. Nothing else enters the score: distance already enters through both heads, so a post-hoc distance multiplier would double-count it and decalibrate the output the notification threshold reads.

One caveat: the second head inherits the same | u sees e selection bias, so its training population is RSVPs the current ranker produced, not RSVPs in general. The mitigation is the same exploration slot, which is why the randomized half of slot 25 has to be logged all the way through to attendance.

Negative sampling

A classifier needs both answers. Impressions that produced no RSVP are the natural negatives. At 857M impressions/day against 1.1M RSVPs, the impression-level positive rate is 0.128%, so negatives outnumber positives about 780:1.

Use impressed-but-not-RSVPed as the primary negatives, then mix in about 20% random candidates that were never impressed at all. The random ones do a job impressions cannot: they teach the model that Tuesday-morning knitting in the next county is not a match. Impressions alone can never teach that, because the current ranker never shows those events.

Injecting those rows carries a correction obligation, and it is the easy one to forget because injection feels like adding realism. Adding never-impressed rows as hard zeros adds negative mass and no positive mass, so every predicted probability drops. The distortion is a constant factor on the odds (p / (1 − p)), not on the probability. At 20% injection a true 0.30 comes out as 0.255, a 15% relative miscalibration landing on exactly the quantity the notification threshold consumes. There is an exact formula that shifts it back, and it must run in the pipeline. Downsample and correct, inject and correct: the same identity with the sign flipped, and a pipeline that does one and not the other is miscalibrated by construction.

Features: geography and time carry the load

With no item history to lean on, the features are the model. The scored row has 134 numbers in it, and 64 of them are the single content-tower vector, so the model the design is really about is 70 hand-built numbers with no learned ID embedding anywhere.

Raw latitude and longitude is a bad feature

A decision tree predicts by asking a chain of yes/no questions. Each question, a split, compares one feature against a threshold, so every split cuts along one axis, never diagonal, never curved. But attendance depends on the difference between two positions, not on either one alone. “Within 5 miles of the user” is a disc, and a tree can only approximate a disc with an axis-aligned staircase: 8 bands cover 79% of it, 16 bands cover 90%. Sixteen bands is not sixteen leaves, since each band costs one y-split plus two x-splits, so it is about 48 leaves for one disc. And the disc moves: a user in Oakland needs a different 48 leaves than one in San Jose, so the leaf budget scales with the number of distinct user neighborhoods and the feature is effectively unlearnable.

The fix is one precomputed number. Haversine distance is the great-circle distance between two latitude-longitude points, over the curved surface of the earth. It collapses two positions into one scalar, and 48 leaves into one split. This is the general rule for any translation-invariant relationship, where shifting both things by the same amount changes nothing so only the gap matters: compute the difference before the model sees it (the same idea as crosses and interactions).

What to use instead

A few terms first. H3 is a global grid that tiles the earth in hexagons at several resolutions, each cell a single integer id; higher resolution means smaller cells. A categorical feature has values that are labels, not magnitudes (cell 8837 is not “bigger” than 8836). Cardinality is how many distinct values a feature can take. Target encoding (TE) replaces a high-cardinality label with the historical outcome rate in that cell, turning the id into a number a tree can split on.

The system uses three H3 resolutions for three jobs, and conflating them is the most common way this design goes wrong:

Coarse cells for crosses (a cross multiplies cardinalities, and fine cells make more combinations than there is data to fill), medium cells for rings (each ring cell costs one index seek), fine cells for travel time (about 5 km is the scale at which one routing estimate is honest about a whole neighborhood). Size a ring in res-5 cells but take it at res 7 and you cover about 7 km instead of ~48, a seven-fold under-fetch that raises no error anywhere; the feed just quietly gets smaller.

Two notes on why H3 is the right shape of grid. Geohash cells (the older, square-celled alternative) are not equal-area: a precision-5 geohash is ~4.9 km wide at the equator but ~2.4 km at 60° latitude, so a “same cell” feature means something different in Oslo than in Nairobi. And a square’s diagonal neighbor is 1.41x farther than its edge neighbor, so “adjacent cell” is an inconsistent unit. H3’s hexagons are near-equal-area with one neighbor distance, which keeps ring queries clean.

Distance decays, but not by any law

Relative RSVP rate against distance, normalized so the 0–2 mile band is 1.00:

miles      0-2    2-5    5-10   10-20   20-40   40+
rel rate   1.00   0.61   0.33   0.14    0.045   0.011

Neither an exponential decay (exp(-d/tau)) nor a power law (d^-a) fits. If either were right, fitting its parameter to each adjacent pair of bands would give the same answer every time; instead the implied tau runs from 4.4 to 20.1 (a 4.6x spread). The reason is that this is not one decay but a mixture of transport modes with thresholds between them: under ~2 miles you walk, 2–10 you drive or take transit, past 20 the event has to be worth a trip. The kinks sit where the mode changes, and the boundaries move from metro to metro, so there is not even one shared set of kinks to fit.

So do not fit a curve. Bucket the distance (boundaries at 2/5/10/20/40 miles, where the kinks are) and let the GBDT learn the step function, and replace distance with travel time wherever you can afford it, because travel time is what is actually comparable across cities. The same 8 miles is 22 minutes in Los Angeles at 2pm Tuesday and 55 minutes at 6pm Friday.

Travel time cannot be computed live: 1,200 QPS peak × 15,000 candidates ≈ 18M route calls per second, which no routing service serves. Precompute a cell-to-cell matrix instead. Per metro: ~970 populated res-7 cells, squared, × 4 dayparts (time-of-day slices, since 8am and 2pm traffic differ) × 2 modes (drive, transit) × 4 bytes ≈ 30 MB. Across 200 metros that is ~6 GB, rebuilt nightly, and it turns a routing call into an array lookup.

Time: memorize the calendar

Hour-of-day as an integer 0–23 is wrong for a model that treats the feature as a magnitude, because hour 23 and hour 0 are one hour apart in reality and 23 units apart numerically. The textbook fix is cyclic encoding: represent the hour by its sine and cosine on a circle, so 23 and 0 land next to each other.

Usually skip it here. What matters is not hour-of-day or day-of-week but their interaction: Friday 8pm and Tuesday 8pm are different products. Bucket on hour-of-week (7 × 24 = 168 buckets) and you have 200M historical RSVPs, about 1.2M per bucket. With that much data in every bucket, a fitted harmonic cannot beat simply averaging what happened. Cyclic encoding is a smoothing prior, and priors are for when you are short of data. Here you are not.

Where you are short of data is the finer interaction that actually matters: metro × category × hour-of-week. Salsa night in Miami on a Friday is a different market from ceramics in Portland on a Tuesday. Slice that finely (200 metros × 30 categories × 168 hours = 1.0M cells) and you have only about 200 RSVPs per cell.

That is thin enough to be noisy but not thin enough to throw away, which is what shrinkage is for: blend a sparse estimate toward a denser, coarser one, in proportion to how little evidence the sparse one has. Shrink up a hierarchy:

cell  ->  metro x hour-of-week  ->  hour-of-week  ->  global

A pseudocount m sets the blend: a level keeps n / (n + m) of its own estimate and gives the rest to its parent. Fit m per level (empirical Bayes, from the ratio of within-cell to between-cell variance) instead of sharing one number, because 200 observations and 3 observations need completely different amounts of help. At m = 20 a 200-observation cell keeps 91% (essentially standing on its own) while the median user’s 3 lifetime RSVPs keep only 3 / (3 + 20) = 13%.

(One unit trap: “200 RSVPs per cell” is a numerator. The n in n / (n + m) is the cell’s trial count, which the volume line does not supply, and the noise on the estimate ranges from 7% to 197% of the rate depending on which reading you take. That spread is exactly why you fit m from measured variance instead of inferring it from a raw count.)

That 13% is the personalization you can actually afford. For the median user the model is predicting the city, not the person, so the honest framing is “learn the city well, then nudge.”

The rest of the feature set

An embedding is a short vector standing in for something unstructured (a title, an image), positioned so similar things get similar vectors; fill rate is the fraction of capacity taken; lead time is the days until the event.

Counting them out gives 134 features, of which 64 are the one precomputed content vector, so the interview-relevant model is 70 hand-built numbers, each a plain decimal or a small integer. That is the cold-start argument restated as a database schema.

The organizer is the item that has a history. An event has 9 interactions; the organizer who runs a monthly series has 400. Every collaborative technique that fails on events works on organizers, series, venues, and categories, so route the collaborative machinery to those persistent entities and let the event inherit. This is the most reusable idea here.

Model choice

There are two stages, and the first is not a model. It is an index lookup, because the constraint that cuts three million events down to one city is geographic, and a geographic constraint has to hold exactly.

flowchart LR
    A["3,000,000 live events<br/>globally"] -->|"geo + time hard filter"| B["~15,000 candidates"]
    B -->|"GBDT ranker over all 15,000"| C["25 ranked events"]

The 15,000 decides the architecture, and it falls out twice. Dividing 3M live events across 200 metros gives 15,000 per metro. And a 25-mile radius (40.23 km) is a disc of π r² ≈ 5,085 km², almost exactly the ~5,000 km² area of a metro. So “events near this user” and “this city’s live catalog” are the same set, and the selectivity (fraction of the catalog surviving the filter) is 15,000 / 3,000,000 = 0.50%.

Why there is no approximate-nearest-neighbor (ANN) index. ANN search finds items whose vectors point roughly the same way as a query’s, trading exactness for speed; HNSW is the dominant version, a layered graph walked greedily toward the query. The usual argument against it here is that filtered HNSW degenerates into a linear scan below ~1% selectivity, and 0.50% is under that line. But you clear that rule of thumb by only 2x, which is not something to build on. The argument that actually holds: an ANN index answers “approximately the nearest,” and geo here is not a preference to approximate, it is a predicate that must be exact. A user shown an event 40 miles outside the radius has been given a wrong answer, not a slightly worse one. Being able to say that is worth more than being able to name three ANN algorithms.

The ranker

• Two-tower runs the user and the item through separate networks, producing a vector each, then scores by multiplying them. That is what makes it fast for retrieval and also what makes it depend on the item having a learnable vector.
• MLP is a plain stack of fully connected layers.
• Wide & deep pairs a memorization component for sparse crosses with a generalizing neural component.

On tabular data (spreadsheet-shaped rows of hand-built numeric features, which is exactly what this feature set produces) trees still beat neural nets, and most relationships here are close to monotone (the answer moves one consistent direction as a feature rises), the shape trees fit most efficiently. A neural component earns its place in exactly one spot: the small text-and-image tower that produces the 64-number content embedding, computed once per event at creation and handed to the GBDT as ordinary features.

Scoring is not free. At 15,000 candidates × 300 trees × depth 8 ≈ 36M comparisons, a vectorized LightGBM at ~1e9 comparisons/second takes about 36 ms. That makes the model the cheap part; assembling the 15,000 feature rows to feed it is the expensive part.

Training

The split must be temporal. A leak is when information that would not exist at prediction time reaches the model during training, so the evaluation flatters it. A random split scatters impressions of the same event across both sides, and fill_rate and rsvp_velocity_6h describe how popular that event became, so a validation row gets scored using knowledge of its own event’s eventual popularity. The gap is large and backwards:

random 80/20 split      offline NDCG@25 = 0.412      online RSVP lift = -1.8%
temporal split          offline NDCG@25 = 0.317      online RSVP lift = +2.4%

The leaky split reported a 30% better model that lost online. Split by time, with the boundary placed before any event that appears in validation was even created.

Class imbalance. Positives are rare (0.128%). You can either weight (tell the loss to count each positive as worth many negatives) or downsample negatives and correct. Prefer weighting, because the weighted objective is exactly the one you wanted with no correction to remember. But 90 days of impressions is ~7.7e10 rows, a genuine compute reason to downsample. The moment you do, an exact analytic correction becomes mandatory, because dropping negatives inflates every predicted probability and the notification threshold reads an absolute number. Weight if you can afford the rows, downsample and correct if you cannot, but never downsample and skip the correction.

Metrics

Offline

The graded scale hides a survey. Grade 3 requires a rating, and only 12% of attendees leave one, so 88% of genuine attendances are capped at grade 2 and the ideal ordering NDCG normalizes against is itself a function of who answered a survey. Read a change in NDCG@25 only against a stable response rate. (NDCG turns a grade into a reward with 2^rel − 1, so 3/2/1/0 becomes 7/3/1/0: a satisfied attendance is worth seven no-shows, about right given a no-show also burns a seat the organizer could have sold.)

Every logged label came from an event the current ranker chose to show, so Recall@25 on that log is recall over the current ranker’s candidate set: the same selection-bias trap. The mitigation is the reserved slot 25, but only half of it counts. On half its impressions it carries a uniformly random event (unbiased, because random selection is unrelated to anything measured); on the other half a cold organizer’s event (selected on “has no history,” exactly the variable under study, so it grades nothing). That leaves 2% of impressions, ~17M/day as clean evaluation data, and it is the only clean data in the system.

Online, and the timing problem

The metric that decides the launch is not observable when the experiment starts. Its duration is set by the calendar, not by statistics: median lead time (7 days) + attendance window (30 days) + novelty burn-in (7 days) ≈ 44 days minimum. Sample size is the easy half: at a baseline of 0.42 attended events per user per 30 days with a standard deviation of 1.1 (over-dispersed, so sd > mean), the standard two-arm formula 16 σ²/δ² at 80% power and a 3% relative MDE needs about 122,000 users per arm, which is nothing against 40M MAU. CUPED (which removes the part of each user’s outcome predictable from their pre-experiment behavior) cuts variance ~35%, buying precision but not a single day.

So the experiment queue is the scarce resource, not traffic: run many arms in parallel instead of trying to make each faster. Randomize on the user, then check for interference, where one arm’s behavior changes the other’s outcomes. Here the mechanism is capacity: an arm that fills a room denies it to control. It is negligible at a 5% allocation and real at a 50/50 launch on a small metro. To measure it, compare a 5% arm against a 50% arm; if the effect shrinks as allocation grows, you have interference.

Serving

One feed request, top to bottom:

flowchart TD
    REQ(["Feed request<br/>user · location · time"]) --> LOC["Resolve location<br/>GPS → H3 res 5 (ring) + res 7 (travel time)<br/>fallback: home cell"]
    LOC --> RING["k-ring 4 around the res-5 cell<br/>61 H3 cells<br/>guarantees 47.6 km ≥ 40.2 km"]
    RING --> IDX[("Geo-time index<br/>cell → events, sorted by start time")]
    IDX --> WIN["Merge in start-time order<br/>exact haversine ≤ 25 mi per id<br/>before any feature fetch"]
    WIN --> EXACT["Stop at now + 21 days<br/>or 15,000 in-radius<br/>whichever comes first"]
    EXACT --> CAND["~15,000 candidates"]
    CAND --> AVAIL{"Hard filters<br/>not sold out · not cancelled<br/>not already RSVPed · not started"}
    AVAIL --> FEAT["Feature fetch<br/>user row · 15,000 event rows<br/>social join · travel-time lookup"]
    FEAT --> RANK["GBDT ranker<br/>300 trees · depth 8 · 36M comparisons"]
    RANK --> CAL["Isotonic calibration, per segment"]
    CAL --> DIV["Diversify (MMR) over category + organizer<br/>cap 2 per organizer"]
    DIV --> EXP{"Exploration slot 25<br/>half random · half cold organizer"}
    EXP --> OUT(["25 events"])
    OUT --> LOG[("Impression log<br/>with the exploration flag")]

The hard-filter step is the one that must be authoritative at request time: a user shown a sold-out event has been given a wrong answer, not a worse one. MMR (maximal marginal relevance) diversifies by repeatedly picking the next item on a trade of its score against how similar it is to what you already picked (here over category and organizer). Only the uniformly random half of slot 25 is unbiased evaluation data; the cold-organizer half is selected on the property being studied.

The geo index is the whole retrieval story

Retrieval is one ring query on one H3 grid, so the design reduces to two integers: which resolution and how many rings. Both are geometry, and both are easy to get wrong in the direction that silently shrinks the product.

The ring must be big enough that every user standing anywhere in their own cell is guaranteed events out to 25 miles, not the average user but the worst-placed one. The obvious formula (k × spacing − apothem) is wrong twice, and both errors shrink the answer, which is the dangerous direction because an under-sized ring raises no alarm:

• The worst-placed user sits at their cell’s far vertex (the circumradius R, 8.54 km at res 5), not its edge (the apothem, 7.40 km). Every H3 table prints the circumradius a line above the apothem, which is how the wrong one gets subtracted.
• The union of k rings of hexagons is not a disc but a notched star. The shortest way out is through the inward notch where two cells meet, where each ring is worth 1.5R = 12.8 km, not the √3·R = 14.8 km center-to-center spacing.

Computing the true guarantee gives (1.5k − 0.5)·R:

              cells      guaranteed radius   vs a 25-mile (40.23 km) request
k = 2           19          22.6 km                    56%
k = 3           37          34.2 km                    85%   <- NOT enough
k = 4           61          47.6 km                   118%   <- covers it

k=3 does not “nearly” cover 25 miles; it covers 85% (or 98% if you derive R from H3’s published area of 252.9 km² instead of its edge of 8.54 km, since the two published averages describe slightly different spherical cells). Both derivations fall short, so round up to k=4. And the missing 15% is not a thin sliver: it is the outer annulus, where most of a disc’s area lives (about 28% of the disc), and it is the suburban band where users have the fewest local options and the largest catchment.

A k-ring is a superset: 61 res-5 cells cover ~11,570 km², more than twice the 5,085 km² disc. That over-read is nearly free and still the right trade. A finer grid (res 6, k=9) would cover less excess area but need 271 cells instead of 61. A seek (one index lookup for one cell’s posting list) costs a round trip regardless of how few ids come back, so the finer grid buys back cheap surplus area and charges you in the expensive currency. A surplus event costs one haversine (35,000 of them is well under a millisecond); a surplus cell costs a seek. So keep the coarse superset and apply the exact predicate after it, on ids, before anything expensive:

61 posting lists, each sorted by start time
   -> merge in start-time order
   -> exact haversine <= 40.23 km, one float per id
   -> stop at 21 days out, OR at 15,000 in-radius candidates
   -> hard filters, authoritative, at request time

The start-time sort makes the candidate count a bound instead of a hope. Truncating soonest-first drops the events furthest in the future, which are the least valuable, so the cap costs nothing in the median metro (which never reaches it) and bites only in dense metros. In a metro at 3x median density, 45,000 in-radius events exist in the window and 15,000 get scored, so 67% of that catalog is never seen. That loss is deliberate and falls on the least valuable events, but Recall@25 counts it identically to an ANN miss, which is exactly why recall is measured at all.

Pricing the ANN alternative. A single global HNSW graph over 3M vectors with geo as a filter fails both ways. Pre-filtering (enforcing geo during the walk) degenerates, because the graph was built over all 3M nodes so the search keeps stepping onto ineligible neighbors. Post-filtering (retrieve then discard out-of-range) fails on arithmetic: retrieve the top 500 by embedding and you expect 500 × 0.005 = 2.5 survivors, so P(zero) = e^-2.5 = 8.2% of feeds are empty, always the same thin metros. To fill 25 slots 99% of the time you would traverse 7,609 nodes of a 3M-node graph to yield 38 in-radius events, where 61 index seeks yield all 15,000 exactly. The geo constraint is not a filter to apply to a similarity search; it is the partition key you split the data by in the first place.

Capacity is a gate, not a feature

Some facts change faster than any feature pipeline can refresh them. Fill rate changes by the minute near a popular event. If capacity lives in the ranker’s feature store (the database of precomputed features, refreshed on a schedule) with a 15-minute refresh, you recommend sold-out events for 15 minutes at a time, and that window lands on exactly the events that sell out fastest, which are the ones the ranker loves most.

So sold-out is a hard filter read from the authoritative counter (the one the booking path itself writes to) at request time, not a feature from a warehouse. The cost is small: one batched Redis MGET of 15,000 keys is about 6 ms, run after the exact haversine cut. Fill rate stays a feature, because 0.6 is genuinely informative; sold out is a boolean gate. The same applies to cancellations and to “the event already started.”

Rollout

The standard sequence is shadow (run the new system on real traffic, serve nothing, compare), canary (serve a small slice), then ramp. The wrinkle is that the decision metric reads out in 5–6 weeks, so a canary that runs for a day has only RSVP rate, which can move opposite to what you sell. The canary therefore gates on safety, not quality: p99 latency, empty-feed rate, click-to-sold-out rate, and the served distance distribution (served, not attended, because attendance is weeks away). Quality waits for the six-week experiment. Canary on one metro, not a traffic slice, which also bounds the capacity interference above. And shadow the geo index separately from the ranker, by replaying a day of requests through the new ring and diffing candidate sets, because a resolution mistake returns a seventh of the candidates it should while every downstream metric stays flat. Candidate-set size is the only place an under-fetch is visible.

Scale and cost

40M MAU · 2 sessions/week · 3 feed loads = 240M requests/week = 397 QPS avg, ~1,200 QPS peak

The per-request budget ends somewhere surprising: fetching features dominates, not the model.

resolve location -> res 5 ring cell + res 7 travel cell   1 ms
geo index: 61 posting-list seeks + start-time merge      22 ms
exact haversine + dedupe over merged ids                  3 ms
hard filters: 15,000-key MGET, pipelined                  6 ms   (network)
feature fetch: 1 user row                                 3 ms   (network)
feature fetch: 15,000 event rows                         25 ms
social join: friends x candidate RSVP lists              20 ms
travel-time lookup: 15,000 array reads                    2 ms
GBDT scoring: 36M comparisons                            36 ms
calibration + MMR + exploration                           8 ms
response: hydrate 25 events, serialize, write            12 ms
                                                       -------
                                                        138 ms  p50
                                                        220 ms  p99

Hydration (fetching the human-facing payload: title, image, venue, price) is small only because it runs on 25 events after ranking instead of on 15,000 before it; hydrating before ranking is the usual way a budget like this breaks. The p50-to-p99 spread is only 1.6x, not the 2.5x you would expect, because the start-time cap fixes work per request; without it, the slowest 1% would be whichever metro is densest (a 3x-density metro would spend 108 ms on scoring alone). 220 ms against a 250 ms deadline is real headroom, but it is 30 ms, not the comfortable 168 ms a design assuming a few thousand candidates would predict.

Sizing the fleet on the 125 ms that is actual CPU: 1,200 QPS × 125 ms = 150 cores busy, × 3 for headroom and failover ≈ 450 cores ≈ 15 servers.

The feature store, the biggest line, is sized by people and their friends, not by events:

40M users x (1 KB profile + 500 seen-event ids x 8 B)     200 GB
social adjacency, 40M x ~150 friends x 8 B                  48 GB
3M live event rows x 2.3 KB                                  6.9 GB
event -> RSVP attendee lists                                 0.8 GB
                                                          --------
                                                           256 GB  (~385 GB with replication)

Events are 7 GB of 256. That is the cold-start argument in gigabytes: the item carries almost nothing, so almost nothing is stored about it, and the two structures that dominate are the user’s own history and the social graph that turned out to be the one surviving collaborative signal. The whole system costs less than two engineers, and the expensive part is the feature store, not the model, which is the normal shape for a tree-based recommender.

Failure modes

Sold-out recommendation

Fill rate is the ranker’s favorite content feature and the countdown to unavailability: the model most loves an event at 0.98 full, minutes from being unshowable, and a stale feature store keeps serving it after it sells out. Detection: click-to-sold-out rate, alerted above 0.5%. Control: the request-time authoritative gate above; also damp fill_rate above 0.9, since the incremental signal up there is mostly “about to disappear.”

The 40-mile drive

A recommendation correct about interest and wrong about follow-through, landing hardest on users with the least data. The distance decay is in the model, but a sparse user’s own travel tolerance is estimated from 2 observations and shrinks to a metro mean that includes people with cars and free Saturdays. Detection: p90 distance of attended events, split by user history depth. Control: make the shrinkage asymmetric, toward the conservative end for sparse users, because an over-long recommendation (a no-show, an empty seat, lost trust) costs more than an under-ambitious one. This is a cost-matrix decision, not a modeling one.

The sparsity of any individual user

Less a failure than a permanent condition: the median user has 3 lifetime RSVPs, the p99 user has 61, and 41% of monthly actives have zero. For most of the audience the model predicts the city, not the person. The feed collapses to popularity (the correct prediction under no information, but it produces a homogeneous feed that generates no new signal), the feedback loop closes, and cold users churn before they generate data. Control: an explicit onboarding step capturing category and neighborhood preferences is worth more than any model change, because it converts 0 observations into ~5 at no latency cost. (30-day attendance for onboarded users is 0.61 vs 0.29, but onboarding is self-selected, so that is a correlation; the causal effect is smaller and needs a holdout.)

Seasonality, which looks like model rot

A model trained in December carries a December expectation of what categories people want into spring, and NDCG@25 slides month over month as holiday markets keep taking impression share after the category is gone. Two changes at once: covariate shift on the item side (the mix being scored changed) and concept drift on the user side (the feature-outcome relationship changed). Detection: PSI (population stability index, one number for how far a distribution has moved) between the category mix impressed and the mix actually live. Control: the hourly p_rsvp retrain handles most of it; add explicit week_of_year and days_to_nearest_holiday features so seasonality has a place to live instead of smearing into category priors.

Recurring events: history you throw away

A weekly run club creates a new event object every Monday. Treated as 52 independent items, each has 9 RSVPs and is permanently cold; treated as one series, it has 470. This is the one place a genuine item embedding is learnable, and a naive schema destroys it. Control: join on series_id (organizer + title similarity + recurrence rule) and inherit attendance rate, rating, and no-show rate from the series. About 22% of events belong to a series and account for 34% of attendance, so this is a third of the problem recovered by a schema decision. Decay inherited signal with a half-life of about 8 instances, so a series that changed venue last month does not carry two years of old ratings at full weight.

Two-sided exposure and the new-organizer trap

A new organizer has no attendance history, so weak features, so a low rank, so no impressions, so no history: a supply-side death spiral nothing breaks from inside. Slot 25 carries two treatments that are not the same experiment: half uniformly random (the unbiased eval log) and half a cold organizer’s event (the supply-ramp intervention). Reserving it costs 2.1% of module RSVPs × (1 − 0.40) = 1.26%, and at 11% of platform RSVPs from the module, 0.14% of platform RSVPs total. That buys a 3x faster supply ramp (median time-to-first-attendee 38 → 12 days) on one half and the only unbiased evaluation data in the system on the other.

Timezones and the bug that ships every year

2025-11-02 01:30 America/New_York occurs twice: clocks go back, so a bare local time is genuinely ambiguous on one weekend a year in every country observing daylight saving. Control: store the UTC instant and the venue’s IANA zone id separately, never a bare local time. Every “is it over / soon / this weekend” predicate runs on UTC; only display and the hour_of_week feature use venue-local time (rendering in a traveling user’s device zone is also a bug).

Summary

Alternatives considered and rejected

Count how many rows point back at the cold-start argument. The item having no history is not one constraint among many; it is the constraint that keeps reappearing.

Conclusion

The whole design follows from one fact: the item is permanently cold, because an event’s interaction history peaks exactly when its value hits zero. Everything else is a consequence.

• Route collaborative signal to entities that persist (organizer, series, venue, category) and let the event inherit. The one collaborative signal that works on the event itself is the social graph crossed with the current RSVP list, because that exists from the first RSVP.
• Split a delayed label into a fast marginal and a slow conditional. p_rsvp retrains hourly to track the catalog; p_attend_given_rsvp retrains weekly on nine-day-old labels, and the product is a calibrated attendance probability. Training directly on RSVP looks like a win and loses 4.8% of attendance.
• Compute invariants before the model sees them. Haversine turns 48 tree leaves into one split; bucketed distance and precomputed travel time beat any fitted decay curve; 168 hour-of-week buckets beat cyclic encoding when you have the data.
• Retrieval is a geometry problem, not a model. Geo is an exact predicate, so partition on an H3 grid (res 5, k-ring 4, 61 cells) instead of filtering an approximate index. Size the ring for the worst-placed user, which means k=4, not k=3.
• Calibration is load-bearing, because the notification threshold reads an absolute probability. Every step that distorts it (negative injection, downsampling) needs its exact correction.
• The binding constraints are the ones you feel late: the decision metric takes six weeks to read out, the latency budget is dominated by feature fetching, not the model, and the feature store is dominated by users and their friends, not events.

One line to remember: the event is permanently cold, so build for the entities that persist and the invariants you can precompute, and let the event inherit the rest.

Further reading

• Uber, H3: A Hexagonal Hierarchical Geospatial Indexing System, h3geo.org. Why hexagons, resolutions, and k-rings.
• J. Friedman, “Greedy Function Approximation: A Gradient Boosting Machine,” Annals of Statistics, 2001. The GBDT foundation.
• G. Ke et al., “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” NeurIPS, 2017.
• Y. Malkov and D. Yashunin, “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs,” IEEE TPAMI, 2018. The HNSW index this design declines to use.
• A. Deng, Y. Xu, R. Kohavi, T. Walker, “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED),” WSDM, 2013.
• A. Niculescu-Mizil and R. Caruana, “Predicting Good Probabilities with Supervised Learning,” ICML, 2005. Isotonic and Platt calibration.

Next: the ad-CTR chapter, where the item has plenty of history, the QPS is a thousand times higher, and the predicted probability gets multiplied by a bid, which is what makes calibration stop being optional.