How to design similar-item recommendations

Scroll to the bottom of any vacation-rental listing and you hit a strip of “similar” places. In this lesson, we’ll build that strip end to end: what “similar” should mean, how to learn it, and how to serve ten bookable results in a few milliseconds. By the end you’ll be able to pick the training signal, draw the negatives that decide whether the model learns anything at all, and defend serving a five-million-item catalogue with no search index.

The design turns on three findings:

• the training signal comes from what travellers browsed, not from what listings look like;
• how you draw the negative examples decides whether the model learns anything useful at all;
• a five-million-item catalogue needs no search index, because the product already partitions it by city.

What the module does

The anchor listing is the one whose page the traveller is reading. The module fills the strip below it.

A market is a city or destination (Lisbon, the Catskills); there are about 1,400 of them. p99 is the 99th percentile: “p99 42,000” means all but the largest 1% of markets hold fewer than 42,000 listings.

This lesson reuses two ideas from earlier chapters, but needs neither in full:

• Two-tower retrieval encodes two sides of a match (a query and an item) into the same vector space so matching becomes a geometry problem. See the video-recommender chapter. Here there is only one tower most of the time.
• A vector index finds the nearest vectors to a query without comparing against all of them. See the RAG chapter. Here the exact comparison is cheap enough that no index is worth building.

Two medians, and why the difference matters

Market sizes are heavy-tailed: the median market holds 900 listings but the mean is 3,600. A handful of huge cities drags the average up. That gives two different medians depending on what you count:

• Line up the 1,400 markets and take the middle one: 900 listings. Most markets are small.
• Line up the 5,000,000 listings and ask how big a market each one sits in, then take the middle listing: it sits in a market of ~15,000. Most listings are not in small markets, because a 380,000-listing city contributes 380,000 entries to this list and a 900-listing town contributes 900.

Both describe the same catalogue; they differ by 17x. The rule for picking between them is to weight by whatever you are counting.

• A training pair is drawn from a listing, so negative sampling has to discriminate against a market of ~15,000 (see negative sampling below).
• A request arrives at a listing (someone reading a page), so the serving scan is also priced at ~15,000, not 900.

So 900 is right for a statement about the catalogue (“most markets are small”) and wrong for anything about traffic. This distinction decides later whether recall@100 measures anything. The hard filters below let ~7.5% of a market through, so the eligible set at the median request is 15,000 x 0.075 = ~1,120 listings, not the 900 x 0.075 = ~68 the wrong median gives. Over ~1,120 candidates, “is the booking in the top 100” asks the embedding to rank it in the top 9%. Over 68 candidates, k = 100 exceeds the whole set, so the metric would be meaningless.

What “similar” means

“Similar” has four defensible meanings, and they build four different systems.

Only substitutability has a booking as its optimum, and it is not recoverable from the listing’s attributes. A concrete case:

Listing A   $210/night, 200 m from B, same photographer, sleeps 2
Listing B   $215/night,                same photographer, sleeps 8
   attribute HIGH   visual HIGH   geo HIGH   substitutability ZERO for a family of six

Listing C   $190/night, 1.2 km from B, different building, sleeps 8
   attribute MEDIUM  visual LOW   geo LOW    substitutability HIGH

One anecdote does not settle a design, so measure it both ways. A decile is a tenth of a sorted population.

pairs in the top decile of VISUAL similarity, ever co-viewed in a session      12%

pairs in the top decile of CO-VIEW frequency ...
   within the same market                                                      88%
   within the same price band                                                  61%
   in the top decile of visual similarity                                      34%

Of the pairs users most often weigh against each other, only 34% look alike. The behavioural relation is not recoverable from the content relation, and that is the empirical reason to train on sessions.

One hard constraint also drops out here: a listing unavailable for the user’s dates has value exactly zero, so it is a filter, not a feature (derived below).

The learning objective

Define substitutability so every term maps to something the logs record. Read P(X | Y) as “the probability of X among the cases where Y held.”

sub(A, B) = P( books B | viewed A this session, did not book A, B was available )

Each condition earns its place:

• “in this session” fixes the trip: same dates, city, party size.
• “did not book A” makes this substitution, not a complement. Booking both is not one standing in for the other.
• “B was available” removes a confound. Otherwise B looks like a poor substitute whenever it was simply booked out.

The learning problem is to produce one embedding v per listing so that substitutable listings sit near each other under cosine. An embedding is a short list of numbers (a vector) attached to each listing, arranged so listings pointing the same way are the ones the system considers alike. The module is then a nearest-neighbour query with constraints attached.

The normalization convention

Two definitions carry the rest of the lesson:

• The cosine between two vectors is their dot product divided by both lengths. It strips out how long they are and leaves only how aligned they are, from -1 (opposite) through 0 (unrelated) to 1 (identical direction).
• L2-normalizing a vector divides it by its own length so the length becomes 1. The result is a unit vector.

For unit vectors, the dot product and the cosine are the same number, so every listing vector that leaves the training job is L2-normalized and the serving dot product is the cosine. The cheap operation the machine runs and the meaningful one you reason about coincide.

There is one exception, and it causes a real bug (see cold start): a weighted average of two unit vectors is not itself a unit vector. Average two arrows pointing different ways and you get a shorter arrow, because the parts that disagree cancel. Anything that blends vectors must re-normalize afterward.

Why calibration does not appear here

In the ad-CTR chapter the model’s output was a probability multiplied by a bid, so calibration was everything: among impressions called 3% likely, 3% must actually convert. That matters whenever something downstream multiplies the number by money.

Here the score is a retrieval key. Only the order matters, and only within one market. Nothing downstream multiplies it, so calibration is irrelevant. Knowing why it mattered before, and that it does not always matter, is the point.

Data: the session is the right unit

A session is an ordered sequence of listing-page views, cut on 30 minutes of inactivity or on a booking. It is a single sitting, not an account’s lifetime.

median views per booking-intent session      28
sessions per month                          300 M
bookings per month                          7.2 M   (2.4% of sessions)

The obvious alternative unit is the user’s whole history: longer and richer. It is the wrong unit, because a user’s trips have different dates, party sizes, budgets and cities, and “alternatives for the same decision” is only true within one trip. The data confirms it directly:

                          same session    same user, different session
P(same market)                0.91                0.34
P(same price band)            0.61                0.29
P(same capacity band)         0.78                0.41

Within a session, intent is roughly constant; across sessions it is not. The session is the largest window over which “these are alternatives for the same decision” holds. A model trained on user-level co-occurrence learns “the same person looked at both,” which is a fact about the person, not the listings.

Two labels: co-view and booking

Within a session the logs offer two kinds of pair, and they are different relations:

• A co-view (A, B) means both were viewed in one session. It says both cleared the user’s filters. Symmetric, and partly a measurement of the search ranker that put them on the same results page.
• A (viewed A, booked B) pair says B dominated A for that trip. Asymmetric, and produced by the user, not the ranker.

Now weigh them. Precision is the fraction of pairs a human rater agrees really are substitutes.

The booking label is more than twice as precise, so why train on views? Volume. Using the population rate of ~10 views per session (not the 28 of a rare booking-intent session), a session of 10 views yields C(10,2) = 45 co-view pairs:

co-view pairs / month   =  300M x 45   =  1.35e10
view->book pairs        =  7.2M x 27   =  1.94e8
                                          69x more co-views, at 2.3x lower precision

Pricing the corpus off the 28-view booking session instead of the 10-view population rate would inflate that ratio eightfold, which is why the distinction matters. 69x the volume at 2.3x lower precision is a trade you take, provided you can still use the rare high-precision signal without diluting it. The architecture below does exactly that.

Inferring the trip

The request carries an anchor and a session, never a booking form. So dates, party size and budget are inferred, in precedence order (take the first present):

The case the design must survive is the ~38% of views with no dates at all (a direct link, a shared URL, an idle browse). The design does not fabricate dates. It degrades the filter: the three-day availability check becomes a single bookable-soon bit: does this listing have at least one free 2-night window in the next 60 days? Its selectivity is ~0.90 against the specific window’s 0.34, so the eligible set widens ~2.6x and the price and capacity filters do the narrowing. Without a trip, “substitutable for this trip” quietly becomes a discovery target, which is the honest thing to serve.

Session embeddings

The model is word2vec, term for term: a session is a sentence and a listing is a word. word2vec learns word vectors from raw text by sliding a window along a sentence and fitting each word to predict its neighbours; words used in similar contexts end up with similar vectors, with nothing labelled by hand.

a sentence:   the   quick   brown   fox   jumped
a session:    L-771 L-8841  L-4127  L-993 L-2264

• A positive pair is two listings within 5 positions of each other in one session (“weighed against each other”).
• A negative pair is a listing paired with one sampled from outside the session (“not weighed against each other”).

Training pulls positives together and pushes negatives apart, producing one d = 32 unit vector per listing.

flowchart TD
    S["Browsing sessions"] --> CV["Co-view pairs<br/>high volume, 0.34 precision"]
    S --> VB["Viewed-then-booked pairs<br/>low volume, 0.78 precision"]
    CV --> SG["Skip-gram training<br/>same-market + global negatives"]
    VB -->|booking as global context| SG
    SG --> EMB["32-dim behavioural embedding<br/>one unit vector per listing"]

The objective

The objective is skip-gram with negative sampling. “Skip-gram” predicts the items around a centre item. “With negative sampling” means the model is not asked to score all 5M listings each step, only to separate one true neighbour from a handful of random impostors.

Written as a loss (a quantity training pushes down) for centre listing c, context listing l in a window of 5, and N = 5 sampled negatives n:

L = - sum over (c, l) [ log sigma(v_c · v_l) + sum over n  log sigma(-v_c · v_n) ]

sigma is the sigmoid 1/(1 + e^-z), which squashes any number into 0–1. In words: for each positive pair push sigma(v_c · v_l) toward 1 (centre and true neighbour agree); for each negative push it toward 0 (centre and impostor disagree). The leading minus turns a log-likelihood you would maximize into a loss to minimize; gradient descent then walks the parameters toward higher likelihood.

The dot product here is a logit on unnormalized training parameters, not a similarity. Normalizing inside the loss would remove the magnitude the optimizer uses to express confidence, so the unit-vector convention applies only to what is written to the index. Normalize once, at export.

The dimension is small on purpose. The useful within-market signal varies along only a few axes: price, capacity, style, sub-neighbourhood. A wider vector spends its spare capacity on memorizing individual listings and on hubness (a few vectors ending up close to everything and turning up in every result list). d = 32 is also what fits the whole index on one machine.

The booking as a global context

The skip-gram window assumes items near each other in the sequence are related. But a booking is the outcome of the entire session: every listing viewed lost to it, at any distance. So for a session that ends in a booking, add the booked listing as a context for every position:

session:  v1 v2 v3 ... v27  BOOK(b)
standard windows:      (v3, v1) (v3, v2) (v3, v4) ...
+ global booking pair:  (v_i, b)   for every i in the session

The relation encoded is “was considered against and lost to,” which is exactly what the module needs to invert. Measured: recall@100 goes from 0.37 (windows only) to 0.44, a 19% relative gain. The rare high-precision signal gets injected as a different kind of pair, not a re-weighted one, so it is not averaged away.

Negative sampling decides the whole model

This is where the model is won or lost. The textbook default draws negatives uniformly from the global catalogue (or in proportion to freq^0.75, which keeps popular items over-represented but less so than raw frequency). Work out what that asks the model to do:

P(a random global negative is same-market) = 15,000 / 5,000,000 = 0.003
P(different market)                        = 0.997

99.7% of negatives are separable by market identity alone. A representation encoding nothing but “which city” already drives sigma(-v_c · v_n) to ~1 on 99.7% of negative terms, so those terms contribute nothing to the loss. Training stops learning anything finer, and “anything finer” is the entire product. The measurement confirms it exactly:

The tutorial default (row 1) is a 4x collapse on the column that matters. Same-market-only (row 2) over-corrects: never having seen a global negative, the model stacks all markets on top of each other and cannot tell Lisbon from Paris. The module’s job is within-market substitution, so weight toward same-market negatives, but keep the global ones for surfaces that need cross-market structure (“consider these nearby towns”). Mix and pick the ratio from the surface you optimize.

What the model actually learns

The nearest neighbours of a $210 two-bedroom in a walkable district, by cosine, before any constraints:

1. $195 two-bedroom, 1.1 km, different host, similar walkability
2. $230 two-bedroom, 0.7 km
3. $185 one-bedroom + sofa bed, 0.9 km
4. $240 two-bedroom, 2.3 km, better transit

No shared photographer, amenity list, or host. The embedding recovered a substitution set, the thing that could not be computed from attributes.

Cold start: the content tower

One population defeats the behavioural model: brand-new listings nobody has browsed. Cold start is the standing name for it. New listings are 8% of live inventory but only 0.4% of session co-occurrences, a 20x under-representation. A listing with no sessions gets a random vector, which is worse than useless: a random point in a space where distance means substitutability.

The fix is a second model, the item content tower, a small network, one tower of the two-tower pattern, mapping a listing’s creation-time attributes into the same 32-number space, trained on the same session objective:

• an H3 cell (a hexagonal geographic tile from Uber’s H3 grid, so a lat/long becomes a categorical feature);
• price bucket, capacity, bedrooms, bathrooms, property type;
• an amenity multi-hot (one on/off flag per amenity);
• instant-book flag and host tenure;
• a photo embedding and a review-text embedding, each produced by a pre-trained encoder used off the shelf.

Retrieval quality for a listing with zero sessions:

The content tower recovers 0.29 / 0.44 = 66% of warm quality on a listing never viewed, and it is twice as good as the averaging heuristic most teams build first. That is why it is trained from the start, not bolted on.

Fading from one vector to the other

Once a new listing gets browsed you have two vectors and must blend them, using shrinkage (pulling a low-data estimate toward a safe default, by an amount that shrinks as data grows):

v = normalize( a * v_behavioural + (1 - a) * v_content ),   a = n / (n + 40)

n is the session count. At n = 0, a = 0 (pure content); as n grows, a climbs toward 1 (behavioural takes over). m = 40 is where the two recall curves cross: below it the behavioural vector is noisier than the content vector, above it better. So a is 0.11 at 5 sessions, 0.50 exactly at the crossover, 0.71 at 100.

Why the normalize is load-bearing

Dropping it manufactures a death spiral out of nothing:

1. A blend of two unit vectors is shorter than either. The shortfall is worst at a = 0.5: two typical vectors (cosine -0.10) blend to length 0.885 at n = 5, 0.672 at n = 40, 0.743 at n = 100.
2. Serving ranks by dot product, which scales with length. An unnormalized blend multiplies a listing’s score by its own norm.
3. So a cold listing pointing in exactly the right direction scores 0.672 where a warm one scores 1.0, a 33% penalty as a pure artifact, peaking right at the crossover.
4. Lower score → fewer impressions → fewer sessions → n stays under 40 → penalty persists.

The worst case names itself. Two antipodal vectors (a listing whose photos say “quiet studio for two” while its sessions say “party house for twelve”) cancel to the zero vector at a = 0.5, which is n = 40. A zero vector has no direction, so a naive “divide by the norm unless it’s zero, then pass it through” hands it straight to the index, where it scores 0.0 against everything forever. A normalization that silently passes a zero vector is not a normalization; the implementation must handle it explicitly.

Business constraints reshape the retrieval design

The embedding gives an ordering; the product’s hard rules decide what may be ordered at all, and they end up dictating the architecture.

An unavailable listing has value exactly zero

Why not make “is it free on these dates” one more feature the model weighs?

click on an AVAILABLE similar listing   ->  8.1% book within the session
click on an UNAVAILABLE listing         ->  "not available for your dates"
                                            23% abandon (baseline abandonment 6%)

The upside of showing an unavailable listing is zero (it cannot be booked). The downside is 23% - 6% = 17 points of extra abandonment. There is no similarity score at which those 17 points are bought back, because the better match still cannot be booked. The model cannot trade off a term whose value is identically zero, so availability is a hard filter, not a feature.

Pre-filter, not post-filter

Two ways to combine similarity search with hard constraints:

• Post-filter: retrieve the most similar, then drop the ones that fail the constraints.
• Pre-filter: narrow to eligible listings first, then search inside them.

To choose, you need each constraint’s selectivity (the fraction it lets through; high means loose). Assuming the constraints are roughly independent, they multiply:

dates available for the 3-night window   0.34
price band                               0.40
capacity >= party size                   0.55
                                         -----
combined                                 0.075

Independence is an assumption doing real work; correlated constraints would multiply to something larger. Post-filtering the top 100 by cosine leaves 100 x 0.075 = 7.5 survivors for a 10-item module, already marginal. Over-retrieving to leave 10 needs k = 400 (with 3x headroom). At the median request, that works.

The tail query breaks it. A traveller with specific needs:

8 guests, pet-friendly, hot tub, peak-season dates
   0.18 x 0.40 x 0.09 x 0.14  =  0.0009 selectivity

post-filter the top 400  ->  E[survivors] = 400 x 0.0009 = 0.36
                             P(module empty) = e^-0.36 = 70%

E[survivors] is the expected count; e^-0.36 is the chance of exactly zero under a Poisson draw (the standard model for a few rare independent successes). Seventy percent of the time this traveller sees an empty module. The users with the most specific needs are the ones post-filtering fails.

The partition is already in the problem

Nobody substitutes Lisbon for Paris. Market is a hard product constraint and a natural shard key (the field you split the data by, so a query only ever touches one piece). Partition the index by market.

This does not change the selectivity: the same 0.075 (or 0.0009 on the tail) applies. It changes the size of the set the filters act on, 5,000,000 / 15,000 = 333x smaller, small enough to scan in full. On the tail query, the market pre-filter scans all ~15,000 and keeps 15,000 x 0.0009 = ~14 survivors instead of 0.36.

Now price that full scan. Each vector is 32 fp32 numbers (four bytes each); a dot product of two is 64 FLOP. The median request scans a market of ~15,000:

bytes    15,000 x 32 x 4 B  =  1.92 MB  @ 100 GB/s  ->  ~19 microseconds
FLOP     15,000 x 64        =  0.96 MFLOP @ 60 GFLOP/s -> ~16 microseconds
                                                          ~35 microseconds total

The largest market on the platform, 380,000 listings, is 48.6 MB and ~24 MFLOP, which comes to ~0.9 ms, a sub-millisecond exact scan. ANN (approximate nearest neighbour) structures only pay once comparing against everything is too slow, above roughly 5M candidates per query; the worst market here is 13x below that. So there is no ANN index in this system. Using the market-weighted median (900) instead of the listing-weighted one (~15,000) would price the scan 17x too low, and the conclusion survives being wrong by two orders of magnitude anyway.

Transpose the calendar and dates become free

A bitmap is one long row of bits, one per listing, where 1 means “qualifies.” Two bitmaps combine with a bitwise AND in a single instruction; one 64-bit AND processes 64 listings at once.

The naive layout stores a 365-bit calendar per listing, so filtering 380,000 listings means 380,000 scattered memory reads. Transpose it: keep one bitmap per day over listings.

NAIVE (calendar per listing)          TRANSPOSED (bitmap per day)
L-0001: 0110111...  (365 bits)        Jul 4: 1011001...  (5M bits)
L-0002: 1001011...  (365 bits)        Jul 5: 1110011...  (5M bits)
   ...  5M rows                          ...  365 rows

Each day’s bitmap is 5,000,000 / 8 = 625 KB; 365 days is 228 MB, fully resident. “Free on Jul 4, 5, 6” is an AND of three 625 KB bitmaps: ~30 microseconds platform-wide, ~2 microseconds inside one market shard. Price band and capacity get the same treatment. The whole constraint set costs microseconds, so availability is a cheap pre-filter, not an expensive post-filter.

Position in the funnel decides the target

“Similar listings” is one of four places the same-looking module could appear. A surface is one placement on one page. Three of them want the model above; the fourth wants its opposite.

After someone books, the one thing they do not want is a substitute for what they just bought. That surface needs a different objective: train on cross-session sequences (the exact user-level signal rejected above), because there “this person also takes ski trips” is the right relation. Same logs, cut at a different boundary.

Metrics

Offline: the recall ladder

Make the offline task the production query: take a held-out session that ended in a booking, use the last listing viewed as the anchor, retrieve k, and check whether the booked listing is in the k. Held-out means those sessions were kept out of training. The k is drawn from the eligible set (~1,120 at the median request), so recall@100 asks the embedding to put the booking in the top 9% of a set the filters already chose. Each row below adds one design decision:

recall@100 on held-out (last-viewed -> booked) pairs
  random within market                              0.021
  attribute nearest neighbour                        0.11
  content tower only                                 0.29
  session skip-gram, global negatives only           0.09   <- market-collapsed
  + same-market negatives, 1:1                       0.37
  + booking as global context                        0.44
  + content blend for items with n < 40 sessions     0.46
  + CSLS hubness correction                          0.48

Check k against the candidate set before quoting it: had the eligible set really been 68 (the wrong, market-weighted median), k = 100 would exceed the whole set and recall@100 would measure the hard filter with the embedding contributing nothing. A recall@k whose k exceeds the candidate set is not a weak metric; it is not a metric at all.

There is a limit to what this ladder can see. The held-out pairs contain only listings the search ranker chose to show, which is already collapsed onto the anchor’s block. So the ladder rewards the geographic collapse the diversity re-rank is built to fix. Measure at the shown slots instead, on both the biased logs and the randomized slot (the only data the logging ranker did not pre-select):

recall@10 of the booked listing        biased logs    randomized slot
  cosine order, no diversity              0.30             0.22
  after MMR + host/300 m caps             0.28             0.29

On biased logs diversity reads as a small loss; on unbiased data it is a clear gain, and the unbiased number is the one that tracks the online +9.4% module booking rate. The ladder is a retrieval diagnostic (did the candidate set contain the answer), not the ship criterion (was the page good).

Online: bookings per user, not module CTR

An A/B test splits users at random into a control arm (old system) and a treatment arm (new), and compares them. The trap here is that the obvious metric moves without anything improving. CTR is click-through rate; cannibalization is a gain taken from elsewhere in the same business, not one newly created.

control:    user browses, books listing A directly.     1 booking
treatment:  user clicks the module, books listing B.    1 booking
module CTR                  0% -> 18%
module-attributed bookings  0 -> 0.09 per session
PLATFORM bookings          unchanged

Both module CTR and attributed bookings call a zero-gain substitution a large win. So the headline is bookings per user, measured on a user-randomized experiment; module CTR is a diagnostic and attribution is a reporting convention.

Sizing that experiment: baseline 0.048 bookings/user per 14 days, per-user standard deviation 0.26 (large, because most users book nothing and a few book several times), and a minimum detectable effect of 1% relative (delta = 0.00048). The standard rule n ≈ 16 sigma^2 / delta^2 gives ~4.7M users per arm. The denominator is squared, so halving the MDE costs four times as many users.

Guardrails and interference

Guardrails you refuse to let get worse; on a two-sided marketplace the supply-side ones matter most, because supply has no vote in the experiment:

• Cancellation rate and mean review score of booked stays: a worse-match substitution that books and then cancels is how a “win” turns into a loss.
• Host exposure Gini and top-1% share of impressions: popularity collapse. The Gini coefficient summarizes inequality: 0 when exposure is equal, 1 when one host gets all of it.
• New-host time-to-first-booking: the supply-side death spiral.
• Median anchor-to-booked distance: geographic collapse.
• Median and p1 eligible-set size: a collapse signals a stale bitmap or a mis-inferred trip narrowing the set to nothing.

One effect is specific to marketplaces: interference, where one arm’s treatment changes the other arm’s outcome. The inventory is consumed: a treatment booking removes a room from the control arm. Negligible at a 5% allocation, real at 50% in a tight market during peak season, so the effect shrinks on full rollout. Detect it by running a 5% arm and a 50% arm at once and comparing effect sizes.

Serving

Everything above assembles into one request path. Solid arrows are the request path (what a traveller waits on); dotted arrows are the offline loop (log becomes pairs, pairs become a retrained model, model becomes new embeddings). The booking write is solid because it must be synchronous.

flowchart TD
    REQ(["Listing page view<br/>anchor + session"]) --> INT["Infer the trip<br/>dates, party size, price band"]
    INT --> SHARD["Resolve market shard<br/>from the anchor"]
    SHARD --> BM["Constraint bitmaps<br/>AND dates, price, capacity<br/>228 MB resident, ~2 us"]
    BM --> SURV["Eligible set<br/>~1,120 at the median request"]
    SURV --> EMB[("Embedding block<br/>32-dim fp32, 640 MB<br/>full replica on every node")]
    EMB --> SCAN["Exact dot products<br/>unit vectors: dot = cosine<br/>no ANN index"]
    SCAN --> CSLS["CSLS hubness correction<br/>2 cos - r_k(x) - r_k(y)"]
    CSLS --> RR["Business re-rank<br/>MMR over distance + price<br/>cap 2 per host, 4 within 300 m"]
    RR --> EXP{"1 slot in 10<br/>reserved for a cold<br/>listing, randomized"}
    EXP --> OUT(["10 listings"])
    OUT --> LOG[("Session log<br/>+ explore flag<br/>+ eligible-set size")]
    LOG -.->|daily| PAIRS[("Pair store<br/>26-week retention")]
    PAIRS -.->|weekly| TRAIN["Skip-gram retrain<br/>26-week window<br/>8-week recency half-life"]
    TRAIN -.->|embeddings| EMB
    BOOK(["Booking write"]) -.->|synchronous bit flip| BM

Two fields hang off the log node. The explore flag marks the one slot in ten that was randomized; it is the join key for the only unbiased evaluation set in the system. The eligible-set size is the request’s post-filter candidate count; it moves up when the no-dates path relaxes the date filter, and down (collapsing) when a bitmap goes stale or an inferred price band is too narrow. It shows a problem before anyone notices an empty module.

The latency budget, and why there is no interesting distributed system here:

infer trip (cached)                    0.10 ms
constraint bitmap ANDs                 0.02 ms
exact dot products                     0.01 ms (p50)  ...  0.20 ms (p99)
CSLS correction                        0.05 ms
business re-rank + MMR                 0.30 ms
                                       -------
measured                       0.5 ms p50,  1.2 ms p99

The p99 gap over the component sum is queueing at peak, not a stage. Against the 80 ms budget, 1.2 ms is ~65x of headroom; nothing here is latency-constrained.

The whole retrieval index is under 1 GB: 640 MB of embeddings (5M x 32 x 4 B) plus 228 MB of calendar bitmaps plus metadata, so every serving node holds a complete replica. That one fact removes sharding, fan-out, scatter-gather tail latency, and any cross-copy consistency protocol. It is a direct consequence of choosing d = 32, and it is the difference between a distributed system and a library.

Cost

Traffic turns into a machine count and a yearly bill. QPS is queries per second.

300M sessions x 10 views x 40% render  =  1.2e9 requests/month
                                       =  460 QPS average, ~1,600 QPS peak
1,600 QPS x 1.2 ms  =  1.9 cores of actual work

The fleet is not sized by compute; two cores would fit on one machine. What sets the count is that each node carries the entire 1 GB index, so a node is a self-contained replica. You size for regional presence and surviving the loss of a region: twelve nodes in each of two regions, 24 machines collectively doing under two cores of work.

The surprising part is that almost none of that buys arithmetic:

• Content tower inference is a reserved GPU idle 99.99% of the time. The real work, 4.8M new/changed listings a year, ~175 GFLOP each (five photos at ~35 GFLOP), is 8.4e17 FLOP, about 0.78 GPU-hours, roughly $2 of arithmetic on a $21,900 reservation. You are buying availability for a job that must fire whenever a host edits a listing, not throughput.
• The retrain fit is free; the shuffle is the job. Five epochs over 8.4e9 tokens is minutes of arithmetic. The eight hours is streaming and shuffling the pair store.
• The serving fleet bills $63.1 k to do under two cores of work. You are paying to have computers standing by.

The genuinely expensive things are off the table entirely: the offline evaluation harness and experiment time. With a 14-day window and 4.7M users per arm, the experiment queue is the bottleneck. Across chapters, the event-recommender chapter cost $230k and was feature-store bound, the ad-CTR chapter cost $5.4M and was latency bound; this one costs $104k and is bound by nothing technical at all.

Failure modes

Each failure gets a mechanism, a detector, and a control.

Hubness and popularity collapse

Why do a handful of listings end up in everyone’s results? Two mechanisms, and one correction fixes both.

1. Gradient volume. A popular listing appears in more sessions, gets more updates, and is pulled toward the centroid (the average position) of many contexts. Its vector ends up near the middle.
2. High-dimensional geometry. In many dimensions, points near the centroid are the nearest neighbour of a disproportionate share of queries. This is hubness, a property of the geometry itself; it would appear even with random vectors.

top 1% of listings by booking volume:
   share of top-10 module slots     34%
   share of actual bookings         11%     ->  3.1x over-exposed

The fix is CSLS (cross-domain similarity local scaling): subtract each point’s local crowding, so a listing in a dense neighbourhood has to be genuinely closer to win.

r_k(z)      = mean cosine from z to its k nearest neighbours (k = 20)
sim'(x, y)  = 2 cos(x, y) - r_k(x) - r_k(y)
            = [cos(x,y) - r_k(x)] + [cos(x,y) - r_k(y)]

Each bracket is “how much closer than usual is this pair, from one side’s view.” A hub sits close to everything, so its r_k is high and it is penalized in proportion to how hub-like it is. r_k is precomputed at index build, so it costs one array read at query time. The result moves accuracy and fairness the same way:

                                 before CSLS   after (k = 20)
top-1% share of top-10 slots         34%           16%
recall@100 on held-out bookings      0.44          0.48

That both improve is rare. The hubs were never being retrieved because they were good matches, so removing that artifact costs nothing and buys both. As a worked example, a $230 two-bedroom can sit second by raw cosine (0.905) yet finish last of four after CSLS, once its own r_k of 0.52 is subtracted, behind a one-bedroom-plus-sofa-bed it beat by 0.021 on cosine.

Here are the vector operations this lesson relies on, gathered in one place; the zero-vector handling is the case a naive implementation gets wrong:

import math

def l2_normalize(v, eps=1e-12):
    # Every vector leaving training is a unit vector, so the serving dot IS cosine.
    n = math.sqrt(sum(x * x for x in v))
    if n <= eps:                       # a zero vector has no direction to rescale
        raise ValueError("cannot L2-normalize a zero-norm vector")
    return [x / n for x in v]

def csls(cos_xy, r_x, r_y):           # r_z precomputed at index build
    return 2.0 * cos_xy - r_x - r_y

def blend(v_behavioural, v_content, n_sessions, m=40.0):
    # a = n/(n+m): m=40 is where the two recall curves cross. Re-normalize, or a
    # convex combination of two unit vectors ships short and scores low.
    a = n_sessions / (n_sessions + m)
    mixed = [a * b + (1.0 - a) * c for b, c in zip(v_behavioural, v_content)]
    if math.sqrt(sum(x * x for x in mixed)) <= 1e-12:
        # antipodal inputs cancel to zero at a=0.5 (n=40); fall back to one side
        mixed = list(v_behavioural) if a >= 0.5 else list(v_content)
    return l2_normalize(mixed)

def eligible(date_bitmaps, price_bitmap, capacity_bitmap, nights):
    # transposed layout: bitwise ANDs over listing-indexed bitmaps, microseconds
    mask = price_bitmap & capacity_bitmap
    for night in nights:
        mask &= date_bitmaps[night]
    return mask

Geographic clustering collapse

The most visible failure: every result is on the same street. Median great-circle distance from anchor to its top-10 neighbours is 180 m. Geography is the strongest predictor of co-view, so the embedding spends most of its capacity reconstructing coordinates and returns a photograph of one block. The user’s actual booking, 1.2 km away, lands at rank 340.

Diversity here is a correction for a representational artifact, not a taste preference, which is what makes it defensible to hard-code. Three fixes, stacked: same-market negatives (reduce the pressure at the source), MMR over (distance, price band, host) in the re-rank, and hard caps of 2 per host and 4 within 300 m. MMR (maximal marginal relevance) greedily picks each next item for a mix of how good it is and how unlike the already-picked items it is. Result: median distance 180 m → 1.4 km, module booking rate +9.4%.

Host flooding and near-duplicates

A property manager with 12 identical units in one building: users compare them constantly, so they co-occur constantly, so their embeddings are nearly identical and occupy the whole top 10. The model is right and the page is useless.

Control: a near-duplicate cluster id keyed on the physical unit, a building geohash (a short string where nearby buildings share a prefix), capacity, and a photo perceptual hash (a fingerprint stable under resize and re-compression), showing one representative per cluster. Keying on host is the tempting mistake: the same apartment listed by two management companies under two host ids would get two keys and never collapse. Host belongs in the exposure cap, not in the identity.

Seasonality

A model of travel behaviour goes stale fast. Trained on Jan–Mar sessions, recall@100 decays by evaluation month: Apr 0.44, May 0.41, Jun 0.36, Jul 0.29, Aug 0.26. Winter co-occurrence structure (ski towns, whole-house rentals, long stays) is not summer’s. Detect with the population stability index (PSI), a single number summarizing how far the recent co-view mix has drifted from the training window’s (see the drift chapter). Control: weekly retrain on a 26-week trailing window with an 8-week recency half-life, plus a same-period-last-year term so seasonal structure is available at the season boundary.

The two-sided fairness loop

A self-reinforcing loop starves new listings: no sessions → content-only embedding (0.29 vs 0.44) → ranks lower → fewer views → still no sessions. Supply is the harder side of the marketplace to acquire, and this quietly suppresses it.

Break it with one reserved slot:

share of module bookings from slot 10          2.4%
cold listing books at                          0.55x a warm one
cost = 2.4% x (1 - 0.55) = 1.08% of module bookings
     x 11% (module's share of platform)        = 0.12% of platform bookings

benefit:  new-host time-to-first-booking  31 d -> 11 d  (2.8x faster)
          new-host 90-day retention       +6 points

The (1 - 0.55) is the loss, not the whole slot: a cold listing still books at 0.55x, so you forfeit 45% of what slot 10 would have earned. That 0.12% of bookings buys a 2.8x faster supply ramp, and the same randomized slot is the only source of unbiased evaluation data in the system. Two payoffs from one slot.

Stale availability

The cheapest possible mistake (refreshing the calendar bitmap on a 5-minute timer) fails hardest on your best output: the module would show listings booked in the last five minutes, which are disproportionately the ones it recommends best. Flip the bit synchronously on the booking write path (it is one bit) and rebuild from the source of truth nightly to catch drift. A fast-moving hard constraint is read from the authoritative store, never from a feature snapshot.

All of it, on one page

Alternatives considered and rejected

HNSW (hierarchical navigable small world) is the most common ANN structure, a layered graph you walk toward the query. A cross-encoder reranker is a second, slower model that reads the anchor and one candidate together, more accurate than comparing two independent vectors, and far more expensive.

Conclusion

• “Similar” means substitutable for this trip. It is the only one of the four candidate meanings whose optimum is a booking, and it is not recoverable from attributes, photos, or distance. That single choice makes the training data sessions, not listing metadata.
• The session is the unit of training, and negative sampling decides the model. Intent is roughly constant within a session (P(same market) 0.91 vs 0.34 across sessions), and drawing negatives from the same market instead of globally is the difference between within-market recall of 0.37 and 0.09; everything else is refinement on top.
• The booking is injected as a different kind of pair, a global context over the whole session (recall 0.37 → 0.44), so a rare high-precision label is used without being diluted into the common one.
• Availability is a filter, not a feature, because its value is identically zero. Transposing the calendar makes that filter cost microseconds.
• The product’s own partition removes the search index. Because nobody substitutes across cities, market is a free shard key; the largest market is a sub-millisecond exact scan, the whole index fits in under 1 GB, and every node holds a full replica. It is a library, not a distributed system.
• The bill is standby capacity, not arithmetic. $104k a year, almost all of it computers waiting, not computing; the real constraint is experiment time.

One line to remember: define “similar” as substitutable for this trip, and the sessions pick the signal, the market picks the shard, and the whole retrieval system collapses into a library.

Further reading

• Grbovic and Cheng, Real-time Personalization using Embeddings for Search Ranking at Airbnb (KDD 2018). The listing-embeddings-from-sessions design this lesson is built on, including the booking-as-global-context and market-negative ideas.
• Mikolov et al., Distributed Representations of Words and Phrases and their Compositionality (2013). The original skip-gram with negative sampling.
• Conneau et al., Word Translation Without Parallel Data (2018). Introduces CSLS as a fix for hubness in embedding retrieval.
• Radovanović, Nanopoulos and Ivanović, Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data (JMLR 2010). Why hubness is a property of high-dimensional geometry.
• Carbonell and Goldstein, The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries (SIGIR 1998). The diversity re-rank.

Next: the feed-ranking chapter, where the inventory is produced by the same people you rank for, so the ranking function becomes an input to next week’s candidate distribution.