How to design video search and ranking

In this lesson, we’ll design a text-query video search engine: it takes a short phrase typed by a person and returns a ranked list of ten videos, each carrying a timestamp to jump to when the query is about a specific moment inside a video. By the end you’ll be able to size each stage from the workload, name the number that gates each design decision, and defend why a video needs machinery a web-page search does not.

This looks like ordinary web search, but it is not. A web page is one document, whereas a video is five documents of very different quality attached to the same identifier.

Those five documents, call them evidence channels, are:

1. The title.
2. The description.
3. The transcript, produced by ASR (automatic speech recognition, a model that turns the spoken audio into text).
4. The on-screen text, produced by OCR (optical character recognition, a model that reads words visible in the picture).
5. The visual track itself: the pixels, with no text at all.

The channels differ in three ways that drive the whole design: how accurate each is, how long after upload it becomes available, and who wrote it. Channels 1 and 2 are written by the creator, whose incentive is your traffic. Channels 3, 4 and 5 are derived from the content itself and are outside the creator’s control. (Tags are a sixth creator-written field; they behave like the title and description and are grouped with them throughout.)

Two terms recur below. NDCG (normalized discounted cumulative gain) is the standard score for how good a ranked list is; it is defined in full where it is first used, under Offline metrics. Recall@k is the fraction of the genuinely relevant items that appear anywhere in the top k results. It answers “was the right answer found at all”, which you have to settle before asking “was it ranked first”. That is exactly why retrieval and ranking need different numbers: retrieval is judged on recall at a large k (did the candidate set contain the answer), ranking on where the answer landed inside that set.

The lesson works in order: frame the problem, choose the metrics, design the architecture, then check it at scale.

The models in this system, and what each is for

“The model” is never singular in a search design. This one has eight distinct learned or measured components.

Offline versus online: where each component runs

Where each one runs decides who pays for it.

• Offline, once per video at upload: ASR, OCR, the visual encoder’s image tower, and the dense encoder’s document side. Their output is written to an index and never recomputed at query time.
• Online, on every query, inside the ~150 ms this system owns: the intent classifier, the query side of both encoders, the L1 ranker, and the L2 cross-encoder. Four components, cheap by construction.

That split is why serving costs roughly eight dollars a day against millions of dollars of one-time indexing, the first thing to say about where the cost is.

The four assumptions holding this design up

All four are checkable, and if one is false a specific part of the design falls over.

1. The creator is adversarial and the content is not. Two of the five channels are written by a party paid in your traffic. The clickbait defense is built entirely on this asymmetry.
2. ASR errors are roughly independent across repeated occurrences of a term. This licenses the 1 - WER^k survival formula. If a decoder mangles a name the same way every time, that formula overstates survival badly.
3. A satisfied click approximates relevance. This is the label under both rankers and both encoders.
4. The dense text bi-encoder is trained on the same click-mined pairs as the visual arm. The least load-bearing of the four: design intent, not measured fact.

Problem framing

Fix the inputs, outputs, corpus size, and latency budget first, then measure how much evidence each channel actually carries, which turns out to decide the entire indexing design.

• In: a text query, typically 1 to 40 characters, plus the searcher’s locale, interface language, and recent history.
• Out: a ranked list of videos, plus a timestamp to jump to when the query is about a moment inside a video.
• Corpus: 500 million eligible videos, mean duration 12 minutes, growing by 500,000 a day. Every later capacity estimate is built from these three numbers.
• Latency: 200 ms at p95 for the whole call. p95 means the slowest 5% of calls may exceed it and the other 95% must not, a promise about the tail, not the average. Of that 200 ms, roughly 150 ms belongs to search and the rest to rendering the page.

What makes it hard: the thing the user wants is usually spoken, sometimes shown, and rarely written, and only the written part is cheap to index.

Sizing the evidence channels

The amount of text per channel differs by two orders of magnitude. Here is one median 12-minute video. The column to watch is unique terms: how many distinct words the channel adds to a keyword index, since a word repeated ten times still creates one dictionary entry.

title             8 words,     8 unique terms      creator-written
description     120 words,    70 unique terms      creator-written
tags             12 words,    12 unique terms      creator-written
ASR transcript 1,800 words,  610 unique terms      content-derived
on-screen OCR    140 words,   90 unique terms      content-derived
visual track     360 sampled frames                content-derived

610 / 8 ≈ 76 and 610 / 70 ≈ 8.7. The transcript carries roughly 75x the lexical surface area of the title and 8.7x that of the description, and it is the only text channel the creator cannot write. That ratio decides the indexing design.

Surface area is not signal — measure the posterior

Having lots of words is not the same as having trustworthy words. What you want is the posterior: the probability a video really is about a term, given that the term appeared in a particular field. That is measured, not assumed, from a 20,000-pair human-labeled sample of (video, term) pairs where a rater answered “is this video substantially about this term?”

Two facts, and between them they are the whole retrieval design:

• Title has the best per-term precision and almost no coverage. Precision is the share of a field’s terms that genuinely describe the video; coverage is how many terms it supplies. The title gets 0.72 of eight terms right: accurate, but eight is nothing.
• Transcript has the coverage and terrible per-occurrence precision. It gets 0.09 of ~430 single-occurrence terms right: mostly noise, but the only field with enough terms to answer a specific query.

So you cannot pick one field, and you cannot treat them alike. A scorer needs two things:

Per-field weights. The natural weight is the log-odds the table already measures: log(p / (1 - p)), evidence strength on a scale where combining two pieces of evidence means adding two numbers. Positive argues for relevance, negative against.

log-odds(title)              = log(0.72 / 0.28)  =  +0.94
log-odds(transcript, k>=5)   = log(0.58 / 0.42)  =  +0.32
log-odds(transcript, k=1)    = log(0.09 / 0.91)  =  -2.31

Because log-odds add, the boost ratio between two fields is exp of their difference: exp(0.94 - 0.32) ≈ 1.9. So the title should be boosted by about a factor of two over a densely repeated transcript term, not ten, not fifty. And a term appearing in a transcript exactly once should count for almost nothing: -2.31 is strongly negative evidence. Seeing a word once in 1,800 spoken words actively argues the video is not about it.

Term-frequency saturation. A rule making the fifth mention worth far less than the first, so a term repeated fifty times does not score fifty times higher. Without it, repetition alone wins.

A formula that does exactly these two things already exists: BM25F (“Best Match 25, Fielded”), the standard lexical relevance formula extended so a document can have several fields of differing trustworthiness. Arriving at it this way matters because the field boosts come out as measured quantities, not tuning knobs. Boosting the title tenfold “because it is the title” is a guess; the log-odds produce the number from measurement.

Which modality answers which query

Knowing which channel answers each kind of query is what justifies spending money on speech recognition and a visual index. Measure the traffic instead of guessing. Six query classes account for the whole mix, from a month of logs hand-labeled on 5,000 queries. The last column is recall@10 using text channels only. Read it as a pass rate, where 0.38 means the system misses nearly two-thirds of the relevant videos.

The transcript-dependent rows sum to 41% + 8% + 6% = 55%. Fifty-five percent of queries are answered by a text channel that did not exist until you ran ASR, and another 14% are answered by a channel with no text at all. That is the business case for both investments, and it sets their very different budgets.

The visual case needs to be made precisely. Take the visual-descriptive slice, a video of a cat knocking a glass off a table:

title        "funny cat compilation #47"
description  "subscribe for more! follow me on ..."
transcript   [laughter]  "oh no"  [laughter]
OCR          (none)

Nobody ever says or writes what is happening. No amount of transcript quality reaches this query, because the information was never encoded in language. That is falsifiable and cheap to check: sample 500 visual-descriptive queries and see whether the relevant videos contain the query terms anywhere in text. Measured, 71% do not. Had it come back at 15%, you would fix the text channels instead of building a visual index. Run the test before spending the money.

The visual arm, stated as a model

“Add CLIP embeddings” names a downloadable checkpoint and skips every decision that matters. State the arm properly.

What it is. A dual encoder: two separate networks whose outputs are designed to be comparable. One is a vision tower: a ViT-B/16 image encoder (ViT is a vision transformer, which cuts an image into fixed 16×16-pixel patches and processes them the way a language model processes words), ~86 million parameters, producing a 512-dimensional embedding per frame. An embedding is just a list of numbers arranged so similar things end up close together. The other is a text tower turning a query into an embedding of the same 512 numbers.

Inputs and outputs. At index time the vision tower turns sampled frames into one 512-d vector each; at serve time the text tower turns the query into one 512-d vector. Similarity is their cosine: the cosine of the angle between the vectors, 1 when they point the same way, 0 when unrelated.

How it is trained, and why that is load-bearing. The two towers are trained contrastively: each batch is built from pairs that belong together (a frame and a phrase describing it), and the objective pushes each frame toward its own caption and away from every other caption in the batch. That objective is InfoNCE, a cross-entropy loss in which the correct partner must beat every other item in the batch, each acting as a negative. A temperature of about 0.07 divides the similarity scores before the softmax, sharpening the contrast so near-misses are punished harder.

The shared space that training produces is the entire mechanism. At serve time the query never touches the vision tower and a frame never touches the text tower, so their cosine is meaningful only because the contrastive objective forced the two spaces to coincide. Train the two encoders independently and you get incomparable spaces, so the arm returns noise. (CLIP is Contrastive Language-Image Pre-training, the published recipe this arm follows.)

Where the training pairs come from. Off-the-shelf CLIP is trained on web alt-text and underperforms on how people phrase video queries, so both towers are fine-tuned on in-domain pairs mined from search logs:

• Strong positives: (frame, query) pairs where the query earned a satisfied click on that video (defined later, under Online metrics).
• Weak positives: (frame, title) and (frame, on-screen text) pairs, plentiful but noisier.

The label is a click, not a human judgement. That is the assumption doing the most work in this arm.

How it is served. Frames are encoded offline, once, so at serve time the arm is a single forward pass of the text tower (~2 ms) plus an approximate-nearest-neighbour (ANN) lookup against the stored frame vectors. An ANN index finds the stored vectors closest to a query vector without comparing against every one.

How you know it works. On the visual-descriptive slice, text-only recall@10 is 0.38; add the visual arm, fuse the two result lists, and it goes to 0.67, recovering most of the 71% of relevant videos that name the answer in no text channel.

The fusion method is RRF (reciprocal rank fusion): score each result by 1/(rank + constant) in each list it appears in, then add. RRF needs only ranks, not comparable scores between lists, which is why it works across arms whose outputs live on different scales.

Routing: a cheap classifier decides which arms fire

You do not run every arm on every query. A tiny linear classifier runs first (4 ms) and turns arms on and off. Its inputs are character n-grams (every run of 3 to 5 consecutive characters, which tolerates typos and needs no fixed vocabulary) plus a detector for quoted spans, since an explicit quotation is almost always a moment query.

flowchart TD
    Q{"Query intent<br/>linear classifier · 4 ms"}
    Q -->|"Navigational · 22%"| N["Lexical only<br/>title/channel boost 3x<br/>skip visual arm"]
    Q -->|"Topical / how-to · 41%"| T["Lexical + dense<br/>transcript-weighted<br/>skip visual arm"]
    Q -->|"Moment / quote · 8%"| M["Lexical + quote index<br/>localization required"]
    Q -->|"Visual-descriptive · 14%"| V["Visual ANN + dense<br/>lexical demoted"]
    Q -->|"Tail / conversational · 6%"| C["Dense-heavy<br/>query broadening on"]
    Q -->|"Entity + attribute · 9%"| E["All arms<br/>fused by RRF"]

    N --> B["Blend · quality prior<br/>diversity · policy"]
    T --> B
    M --> B
    V --> B
    C --> B
    E --> B
    B --> R(["Ranked results"])

The two lanes that pay for a channel this design spends real money building are topical/how-to (the 41% that ASR exists for) and visual-descriptive (the 14% that the visual index exists for). The other four are served by evidence the system would have had anyway.

Each lane is a different answer to “which evidence do I trust for this query”:

• Navigational (22%): naming a video you already know exists. Lexical only, title and channel boosted threefold, visual arm skipped.
• Topical / how-to (41%): answered by what someone said. Lexical arm weighted toward the transcript, plus the dense text arm. Visual arm skipped.
• Moment / quote (8%): adds the quote index and requires localization. Returning the right video without a timestamp is a failure for this class.
• Visual-descriptive (14%): fires the visual ANN and dense arms, with the lexical arm demoted but not removed, because the video’s words are usually misleading (“funny cat compilation #47”) but occasionally right.
• Tail / conversational (6%): leans on the dense arm and switches on query broadening, relaxing the requirement that every query term match.
• Entity + attribute (9%): no single channel suffices, so all four arms fire and their lists are combined by RRF.

All six converge on one blend stage, which applies a per-video quality prior (how good this video is independent of any query), diversity rules that stop one channel filling the page, and a policy filter.

Because the entity+attribute lane fires everything, two arms run on more traffic than their headline class: the visual arm on 14% + 9% = 23%, the quote index on 8% + 9% = 17%. Both are affordable at those rates and prohibitive at 100%. That is the general lever: the expensive path is paid for by the classifier that keeps it off the common path.

Temporal granularity: whole video or 30-second slice?

What is a “document” in a video index? The tempting design is to index 30-second segments, because it gives deep links: a result that drops the user at 8:32 instead of at the start. Deep links help, so the question is what they cost, and both sides must be priced.

What segmentation buys. On moment queries, landing at the start of a 12-minute video forces the user to scrub for the part they wanted. Measured, that is expensive:

moment query, land at t=0:      abandon within 20 s  38%   satisfied-click rate  0.41
moment query, land at t=8:32:   abandon within 20 s  11%   satisfied-click rate  0.69

What segmentation costs, and it is not index size. The reflex objection is “that is 24x the documents” (720 / 30 = 24 slices where there was one). That is nearly wrong. A lexical index is a set of postings: one entry per (term, document) occurrence. The posting count is unchanged by segmentation. The same 1,800 transcript tokens get posted either way, just against 24 identifiers instead of 1. The dictionary and skip lists grow, but compressed postings dominate, so the real increase is about 1.35x, not 24x.

The real cost is evidence fragmentation, a recall cost. Trace one query:

query "civic rear brake pad torque spec"      5 content terms
video-level:   all 5 terms appear somewhere in the 1,800-word transcript
               -> conjunctive match, strong BM25F score
segment-level: "civic" at 0:14, "brake pad" at 4:02, "torque spec" at 9:41
               -> no 30-second window contains all five
               -> every segment is a weak partial match

A conjunctive match (every query term present in one document) is what a strong lexical score requires, and segmenting makes it impossible for long queries by construction.

At 4+ terms the drop is 0.77 → 0.41. Segmenting loses nearly half the relevant videos, and the recall it destroys is concentrated in exactly the long, specific queries that were the reason you built search. Overlap softens it (0.41 → 0.55) but cannot fix it: a 4-term query spread over 9 minutes fits no window short enough to be a useful deep link.

The resolution: localize at rank time, not index time

Move localization out of indexing and into the ranking stage, where you are looking at 100 documents instead of 500 million. Retrieve against whole videos, so no recall is lost. Run the L1 ranker (the cheap first pass, ~1,700 → 100). Then, for those 100 videos only, fetch each video’s segment payload and score its segments against the query you already have.

segment payload, per video:  24 segments × (start_ms, 40 top terms + positions)
                             ≈ 2.1 KB compressed
500 M videos × 2.1 KB        =  1.05 TB   keyed by video id, not searchable
fetch 100 payloads at rank time, 8-way parallel  ≈ 18 ms

You get the deep link without the recall loss, because the segment structure is a payload, not an index. A payload is data fetched when you already know the key; an index is data you search by content. The 1.05 TB is keyed by video id and never searched, which is why it costs recall nothing.

The one case this misses is a moment query whose terms exist only inside one segment and are too weak to retrieve the whole video: quotations, mostly. Those get a genuine segment-level index restricted to high-IDF n-grams. IDF (inverse document frequency) measures how rare and therefore informative a term is, so a high-IDF n-gram is an uncommon phrase. Restricting to quoted spans of four or more tokens keeps it at roughly 3% of a full segment index’s postings.

Retrieval architecture

The pieces assemble into a serving path: four retrieval arms produce candidates, two ranking models narrow them, and a millisecond budget forces the shape. Follow the candidate count down the diagram. Four arms fan out in parallel, then every stage after the merge narrows: ~1,700 → 100 → 10.

flowchart TD
    Q(["Query"]) --> QU["Query understanding<br/>spell · segmentation<br/>intent class · language<br/>8 ms"]
    QU --> LEX["Lexical BM25F<br/>title · desc · tags · transcript · OCR<br/>12 shards · 25 ms · widest fan-out"]
    QU --> DEN["Dense transcript ANN<br/>768-d text emb · HNSW<br/>12 ms"]
    QU --> VIS["Visual ANN<br/>pooled frame emb<br/>fires on 23% · 9 ms"]
    QU --> QUO["Quote index<br/>4-gram spans<br/>fires on 17%"]

    LEX --> MRG["Merge · dedupe by<br/>content hash + channel cap<br/>~1,700 candidates · 3 ms"]
    DEN --> MRG
    VIS --> MRG
    QUO --> MRG

    MRG --> L1["L1 ranker · GBDT<br/>190 cheap features<br/>biggest cut: 1,700 -> 100 · 15 ms"]
    L1 --> SEG["Segment payload fetch<br/>+ localization scoring<br/>I/O-bound: 100 reads · 18 ms"]
    SEG --> L2["L2 cross-encoder<br/>query × title+best segments<br/>GPU-bound: 6 layers · 100 pairs · 11 ms"]
    L2 --> BLEND["Blend · quality prior<br/>freshness · diversity<br/>policy filter · 4 ms"]
    BLEND --> OUT(["10 results<br/>+ deep-link timestamps"])

Two stages sit next to each other and are near opposites. Segment fetch is the only I/O-bound stage: 18 ms waiting on 100 key-value reads, almost no arithmetic. L2 is the only GPU-bound stage: 603 GFLOP of dense matrix work, almost no waiting. The lever that shortens the fetch (more parallel reads) does nothing for L2, and the lever that shortens L2 (a bigger GPU) does nothing for the fetch. Confusing the two is how teams buy the wrong hardware.

Walking the serving path

Query understanding (8 ms) does four cheap things: spelling correction; segmentation (splitting brakepads into two words); intent classification; and language detection.

Four retrieval arms run in parallel, so the group costs its slowest member, not the sum.

• Lexical BM25F (25 ms) scores the query against all five fields across 12 shards (the index cut into 12 pieces on 12 machines, queried at once), so the arm is as slow as its slowest shard.
• Dense transcript arm (12 ms) looks up the query’s 768-d text embedding in an HNSW index. HNSW (hierarchical navigable small world) stores vectors as a graph with long-range links at the top and short-range links at the bottom, so a search hops coarsely toward the right region then refines. Its ef_search parameter (200 here) is how many candidates the search keeps alive while walking the graph. Larger is more accurate and slower.
• Visual arm and quote index fire only when the classifier says so, on 23% and 17% of traffic.

Merge (3 ms) removes duplicates by content hash (so a re-upload does not take two slots) and applies a channel cap (so one prolific creator cannot fill the page). About 1,700 candidates survive.

L1 (15 ms) is a GBDT scoring all 1,700 on 190 deliberately cheap features (field match counts, video age, click priors), all already in memory, none requiring a document fetch. It keeps the best 100.

Segment payload fetch (18 ms) happens only now, for those 100, and produces the deep-link timestamp.

L2 (11 ms) re-scores the 100 with a cross-encoder, then blend (4 ms) applies the quality prior, freshness, diversity, and policy filtering before ten results go out.

The latency budget

The key is which lines are parallel (only the largest counts) and which are serial (they all add). The three extra retrieval arms are parallel with lexical; everything else is serial.

query understanding                                8 ms
lexical retrieval, 12 shards in parallel          25 ms   <- dense (12), visual (9) hide inside this
merge + dedupe                                     3 ms
L1 GBDT, 1,700 × 190 features                      15 ms
segment payload fetch + localization              18 ms
L2 cross-encoder                                   11 ms
blend + policy                                      4 ms
                                                  ------
                          serial path, p50         84 ms
                                          p95      140 ms

The 84 ms is the serial path (8 + 25 + 3 + 15 + 18 + 11 + 4); the dense and visual arms hide inside the lexical arm’s 25 ms. That leaves 66 ms of headroom against the 150 ms budget, which absorbs the tail: a slow shard, a cold page cache, a GPU queued behind another request. Hence p95 at 140 ms, not 84.

The two rankers

Both are trained on the same labels (satisfied clicks, corrected for position bias), which is what makes the pair coherent: both approximate the same target and differ only in how much evidence each looks at. L1 takes cheap in-memory features for ~1,700 candidates; L2 takes the actual text of 100 (query, document) pairs. Both run online, on every query. L2 is judged by NDCG@10. L1 is judged differently: whether the 100 it keeps still contain what L2 would rank on top, the only way to catch L1 quietly throwing away the right answer before L2 sees it.

Deriving L2’s 11 ms

L2 is a cross-encoder. The bi-encoder arms embed query and document separately, so the document side can be precomputed offline. A cross-encoder concatenates query and document into one sequence and runs a transformer over both, so every query word can attend to every document word, far more accurate, and far more expensive because nothing can be precomputed.

Configuration: 6 layers, hidden width d = 384, T = 256 tokens per pair, 100 pairs per request. The cost has two parts: the matrix multiplies inside each layer, which grow linearly with sequence length, and the attention comparisons between tokens, which grow with T^2 because attention compares every token with every other.

matmul term    ≈ 543 GFLOP
attention term ≈  60 GFLOP   (only ~10% here at T=256; grows with T^2)
               ---------
total          ≈ 603 GFLOP
on one A10G at ~60 TFLOP/s effective  ->  ~10.1 ms

An A10G is a mid-range inference GPU; “60 TFLOP/s effective” is what real workloads actually achieve, well below datasheet peak. That 10.1 ms is where the 11 ms budget line comes from.

The cross-encoder is affordable only because it runs on 100 documents. Run it over all 1,700 merged candidates and it is 603 GFLOP × 17 ≈ 10.3 TFLOP, about 172 ms, more than the entire 150 ms budget, spent on one stage. That is the two-stage argument in its cheapest form: a cheap model from thousands to hundreds, an expensive model to order the hundreds.

Why lexical retrieval does not go away

Why keep a keyword index alongside two embedding arms? This system has two dense arms and they are not the same. This one is a text bi-encoder over the transcript and title, one encoder applied twice and independently, once to the document (offline) and once to the query, matched by cosine. It has never seen a pixel; the visual dual encoder is the only arm that touches frames.

Dense-only retrieval is tempting and wrong here, for a reason specific to video:

query   "torx t25 vs t27"
dense   returns general "screwdriver bits" and "socket set" videos — a lossy
        768-d compression of a 1,800-word transcript keeps topic, drops one digit
lexical returns the video whose transcript says "t27" eleven times

Long-tail exact tokens (part numbers, model years, error codes, proper nouns) are precisely what an embedding compresses away, and they are over-represented in video search because video is where people go for repair, gaming, and product content. Hybrid retrieval is not hedging; the two arms answer disjoint query classes. Fuse with RRF instead of trying to calibrate a BM25 score against a cosine, since those numbers live on incomparable scales and any linear blend is a hyperparameter you retune every index build.

ASR: the highest-value channel, and how it fails

ASR answers most of the traffic and deserves three hard looks: what it costs, why its headline accuracy metric is wrong for search, and how its errors land unevenly across speakers.

Cost

A GPU-hour is one hour of one GPU. “120x realtime” means one GPU transcribes 120 hours of audio per hour of wall-clock time, achievable only because the work is batched so the hardware is never idle. The distilled ASR model (one trained to imitate a larger, more accurate model at a fraction of the cost) runs at about 120x on a batched offline pipeline.

backfill (one-time): 500 M × 12 min ≈ 100 M audio-hours
                     100 M / 120x    ≈ 833,000 GPU-hours  ×  $2.50  =  $2.08 M
steady state: 500 k new videos/day ≈ 100,000 audio-hours/day
                     ≈ 833 GPU-hours/day × $2.50 = $2,083/day = $0.0042/video

The backfill is a budget line item; the steady state is a rounding error (under half a cent per video). So the decision is “do it once, properly”, not “roll it out gradually to control cost”.

Two things drive that $2.08 M and both are assumptions, not FLOP counts. The 833,000 GPU-hours comes entirely from the 120x throughput figure, the number to challenge, not any arithmetic rate. And $2.50 per GPU-hour is deliberately not the serving rate: it is a training-grade accelerator run as an offline batch job, whereas serving is priced on an A10G at $0.75/hr. The backfill wants throughput and can afford the expensive card because it runs once; serving wants cost per query and takes the cheap card.

One sensitivity worth carrying: unbatched, the same model runs at ~20x instead of 120x, so 100 M / 20 = 5.0 M GPU-hours ≈ $12.5 M, about 6x the batched cost. Batching is a 6x lever on an eight-figure number, so it belongs in the answer, not a footnote.

Why WER does not translate linearly into retrieval loss

WER (word error rate) is the fraction of spoken words the transcript gets wrong, counting substitutions, deletions and insertions. An aggregate WER of 12% sounds like it should cost 12% of retrieval. It costs far less on most terms and far more on the ones that matter, and the mechanism is repetition.

A topical term appears k times, and the transcript only has to get it right once for the term to be searchable. One correct posting is enough. So the term is lost only if ASR botches every occurrence. If errors are independent across occurrences:

P(term survives)  =  1 - WER_term^k

Independence is the load-bearing assumption: it fails for a decoder that mishears the same name identically every time, in which case survival is closer to 1 - WER regardless of k and everything below is optimistic.

Nine repetitions at a 7% error rate makes loss essentially impossible (1 - 0.07^9 ≈ 1.0). One utterance at a 41% error rate makes loss a coin flip (1 - 0.41 = 0.59). The two effects compound instead of cancelling, because the terms with high error rates are also the terms said only once or twice.

Repetition rescues exactly the low-IDF terms you did not need and abandons exactly the high-IDF terms the query depends on. A query is usually anchored by one rare term, and losing that anchor means the video is never retrieved at all. The failure is total, not graded. You get nothing, not a slightly worse ranking.

So aggregate WER is the wrong ASR metric for search. The right one is entity WER at the video level: the fraction of videos where at least one named entity present in the audio is absent from the transcript. It is 0.37 for this corpus (the share-weighted average of the accent table below), and it is the number the search team should report back to the speech team. (An aggregate that lands below every group it averages, here below the best group’s 0.26, is a reporting bug.)

Three fixes, cheapest first

All three reduce entity loss without training a better ASR model.

a) Contextual biasing. The title, description, tags, and the channel’s past vocabulary are all available before the audio is decoded. Feeding them to the decoder as a shallow-fusion bias (nudging the probability of each candidate word at each step, without retraining) costs about 2% of ASR throughput and cuts entity WER by roughly 38% relative: 0.34 → 0.21 on product and model names, 0.37 → 0.23 at the corpus level. The creator already told you the hard words; the decoder just was not listening.

b) Lattice or n-best indexing. An ASR decoder produces a ranked set of competing hypotheses (an “n-best list”, or a lattice if you keep the graph of alternatives) and throws all but the top one away. Index the top three hypotheses per span instead, weighted by decoder confidence. Transcript postings grow 2.4x and entity recall@100 rises 6.1 points, worth it unless transcript postings are already the storage bottleneck.

c) Phonetic fallback in the query path. Store a double-metaphone key alongside each term (a short code capturing roughly how a word sounds instead of how it is spelled, so similar-sounding words collide on purpose) and when a query returns nothing, retry against those keys. This catches a user typing navara against a transcript that heard navarro. Index cost is near zero, and because the retry only fires on the zero-result path it cannot damage precision on working queries.

The accent problem is a fairness problem

The errors above are not spread evenly across speakers, and an accuracy gap in a speech model becomes an income gap for a group of creators. AAVE below is African-American Vernacular English, a dialect general-purpose speech models are consistently worse at.

speaker accent group     WER     entity WER    share of corpus
US general               0.081      0.26           41%
UK / IE                  0.094      0.29           11%
Indian English           0.163      0.44           14%
Nigerian English         0.178      0.47            4%
AAVE-leaning             0.171      0.45            7%
Non-native, other        0.192      0.51           23%

The raw gap is 0.192 / 0.081 = 2.4x. The compounding is worse than the ratio, because of the 1 - WER^k curve: at k = 1 (a name said once, the common case) entity loss goes from 0.26 to 0.51. More than half the named entities in an affected creator’s video are not searchable.

This will not show in any aggregate metric. Those creators are a minority of the corpus, so aggregate NDCG moves about 0.3 points, indistinguishable from noise. Report retrieval recall by speaker-accent cluster as a standing metric, in the same table as aggregate NDCG, or the regression is invisible by construction.

Offline metrics

Start with what you can compute without shipping. The primary metric is NDCG@10 (normalized discounted cumulative gain at 10), whose four words are four steps:

1. Gain: each result scores by how relevant a human rater graded it, here 0 to 4.
2. Discounted: divide each gain by a discount growing with position, so a hit at rank 1 counts more than the same hit at rank 9.
3. Cumulative: sum the discounted gains over the top ten.
4. Normalized: divide by what the best possible ordering of the same results would have scored.

The output is 1.0 for a perfect page and near 0 for a useless one. That fourth step is where two traps live (worked out later under “the long tail with no good result”).

No single metric here is safe to gate on alone. Each row’s trap is the reason the row below it exists:

(MRR is mean reciprocal rank, the average of 1/rank of the first correct result.)

Two things about NDCG are specific to video. Graded relevance must include a duration-aware grade: a 3-hour livestream that contains the answer at 1:47:00 is not the same result as a 4-minute video about exactly that. If the rubric does not say so, raters grade on topicality alone and NDCG promotes the livestream. Add “effort to reach the answer” explicitly. And report stage-wise attribution: recall@1000 (retrieval), NDCG@10 given the retrieved set (ranking), and end-to-end NDCG@10. If end-to-end is flat while ranking-given-retrieval improved, retrieval regressed and the ranker absorbed it, a common failure a single number hides.

Online metrics, A/B, and position bias

Every online metric is a definition you choose, not a quantity waiting in the logs. Write the definitions down first, because most disagreements about the numbers are really disagreements about the definition.

click                 any result click
satisfied click       click AND (watch >= 30 s OR watch >= 30% of duration)
                      AND no return-to-results within 60 s
good abandonment      no click, no reformulation, no re-query in 10 min
                      -> answer was in the snippet, or the user gave up; ambiguous
reformulation         a new query within 60 s sharing >= 1 content term

The “satisfied click” definition joins three conditions with AND; the third (no return to the results page) is what separates a good result from a misleading thumbnail. Satisfied-click rate is the primary metric; reformulation rate is the sharpest negative one. Raw CTR (click-through rate, the share of impressions that get any click) is actively misleading here, because clickbait raises it by construction, so CTR stays a diagnostic and never becomes a goal. Total watch time from search rewards long videos mechanically, so use watch fraction thresholds for satisfaction and keep total watch time as a guardrail only.

Position bias, and why it makes A/B underpowered

An observed click is not evidence of relevance on its own. It is relevance gated by whether the user looked at that rank at all, and the two factors multiply:

P(click | result r, rank j)  =  examination(j) × relevance(r)

examination(j) is the probability the user’s eye reached rank j; relevance(r) is the probability they would click if they saw it. You observe only the product. The examination term is measurable by a cheap experiment. Swap the results at two positions for 1% of traffic and watch whether the click rate follows the position or the result; whatever follows the position is examination:

rank   1      2      3      4      5      6      8     10
exam  0.72   0.51   0.39   0.31   0.26   0.22   0.17  0.13

Rank 1 is examined 0.72 of the time and rank 10 only 0.13, a 5.5x falloff. Two consequences follow, and together they are why ranking experiments are not run as plain A/B tests.

Most click mass is decided by rank 1. Two systems that differ only on ranks 3 through 10 produce nearly identical click rates, because those ranks are barely examined. The standard sample-size formula for comparing two proportions is n ≈ 16 · p(1-p) / delta^2 (the 16 encodes a 5% false-positive rate and 80% power). At a baseline satisfied-click rate p = 0.44:

• A big 0.5-point effect needs ~158,000 queries per arm, reads out in a day.
• The 0.1-point effect a ranks-3-to-10 change actually produces needs ~3.9 M queries per arm, since sample size scales with 1/delta^2. That is a week per experiment, fighting seasonality.

Interleaving removes the confound instead of estimating it. Team-draft interleaving builds one result list by alternating picks from the two rankers, then attributes each click to whichever ranker contributed that result. Because both rankers contribute results at comparable positions, examination(j) is the same for both and cancels. You never estimate it. The same 0.1-point decision now needs about 6,000 to 15,000 sessions, a 260x to 650x speedup. (The often-quoted “10 to 100x” is the speedup on the easier 0.5-point decision, where an A/B was already affordable. It understates the real case by an order of magnitude.)

Two functions implement it. The first builds the interleaved list; the coin flip at the top of each round is what keeps the position distributions fair. The second decides who won a session. And sessions, never individual clicks, are the unit of inference, because counting a six-click session as six observations inflates significance.

import random


def team_draft(list_a, list_b, k=10, rng=random.Random(0)):
    """Interleave two ranked lists and record which ranker contributed each slot.

    Both rankers end up in comparable position distributions, so the
    examination term cancels in the click comparison instead of being
    estimated. The per-round coin flip is what makes the positions
    exchangeable; without it list A wins rank 1 every time and you have
    rebuilt the bias you were removing.
    """
    cursor = {"A": 0, "B": 0}
    lists = {"A": list_a, "B": list_b}
    seen, blend, credit = set(), [], []
    while len(blend) < k and any(cursor[n] < len(lists[n]) for n in lists):
        order = ["A", "B"] if rng.random() < 0.5 else ["B", "A"]
        for name in order:
            src, i = lists[name], cursor[name]
            while i < len(src) and src[i] in seen:
                i += 1
            if i < len(src) and len(blend) < k:
                seen.add(src[i]); blend.append(src[i]); credit.append(name); i += 1
            cursor[name] = i
    return blend, credit


def session_outcome(credit, clicked_ranks):
    """A session votes for A, for B, or ties. Sessions are the unit of
    inference, never clicks: counting a six-click session as six observations
    inflates significance."""
    a = sum(1 for r in clicked_ranks if credit[r] == "A")
    b = sum(1 for r in clicked_ranks if credit[r] == "B")
    return (a > b) - (a < b)

Interleaving is the right default for ranking changes and the wrong tool for anything else. It cannot measure a UI change, a change in the number of results, or long-term effects, because both arms live inside the same session. Use it to get from 40 candidate rankers to 3, then run a ship-decision A/B on the finalists.

Scale and cost

Size what the system keeps in memory and what it burns per query. The two are not the same order of magnitude, which changes what to optimize.

Index sizes

Three compression terms: delta + varint stores the difference between consecutive document ids and encodes each difference in as few bytes as it needs; fp16 stores each number in 16 bits and int8 in 8, so an int8 vector is a quarter the size of a 32-bit one.

The lexical tier (everything keyword search needs) at ~3.6 bytes per compressed posting:

transcript postings   500 M × 1,800 tokens, compressed  =  3.24 TB
title/desc/tags/OCR   500 M × 280 tokens                =  0.50 TB
segment payloads      500 M × 2.1 KB                     =  1.05 TB
quote index, 4-gram spans, high-IDF only                =  0.31 TB
                                                           --------
lexical tier                                             ≈  5.10 TB
(n-best transcript expansion, ×2.4, would add +4.54 TB — a proposal, not shipped)

The vector tier (everything the two embedding arms need):

dense text embeddings    500 M × 768-d × 2 B (fp16)         =  768 GB
HNSW graph overhead      fixed ~276 B/vector at M=32        ≈  0.14 TB (+18%)
visual pooled embeddings 500 M × 512-d × 1 B (int8)         =  256 GB
                                                              --------
vector tier                                                 ≈  1.16 TB

The HNSW line contradicts a rule of thumb worth correcting. M = 32 is how many neighbour links HNSW keeps per vector, and that link block is a fixed ~276 bytes no matter how long the vector is. So the links double memory only when the payload is about 276 bytes, roughly the case at 128 dimensions in fp16. At 768 dimensions the payload is 768 × 2 = 1,536 bytes and 276 / 1,536 ≈ 18%. Copying “HNSW roughly doubles memory” across designs is how a 0.9 TB tier turns into a 1.5 TB budget request.

Sharding cuts one logical index into pieces on separate machines, 12 lexical shards here, 5.10 TB / 12 = 425 GB each, with each shard’s hot subset resident in the operating system’s page cache. The split is by a hash of the video id, not by topic: a query does not know its topic in advance so it would fan out to every topic shard anyway, and grouping a topic on one machine creates a hotspot the moment that topic is popular. Hashing spreads storage and load evenly by construction.

Frame embeddings: the number that kills naive designs

The most useful thing to derive here is the design you did not build. These are four choices of how finely to store visual embeddings, at ~1 KB per 512-d fp16 vector, over 500 M videos:

1 frame / 2 s               360 frames/video   ->  180 TB
shot keyframes, shot 4.2 s  171 frames/video   ->   86 TB
30-second pooled segments    24 per video      ->   12 TB
video-level pooled            1 per video      ->  500 GB

The ratio between the ends is 180 TB / 500 GB = 360x, two and a half orders of magnitude, not the five people reach for when they call it “impossible”. Frame-level indexing is off by 360x, so the resident visual index is the video-level pooled vector, the 256 GB int8 line above (500 GB in fp16 for comparability, 256 GB stored as int8: same vectors, half the precision, half the bytes).

But mean-pooling averages every frame into one vector, so a brief event is averaged into invisibility. A glass leaving a table lasts ~2 seconds, which is 1 of the 360 sampled frames, so it contributes 1/360 of the pooled vector. Pool to 30-second segments instead and it is 1 of 15 frames, a 360/15 = 24x stronger signal. That dilution, not impression volume, is why the 30-second segment tier exists.

The 12 TB line is what segmenting everything would cost, and that is not built. The tier is built lazily, only for the visual-descriptive slice, over the top ~40 M videos by impression volume (how often a video is shown). That is 8% of videos and covers 91% of visual-query traffic: 40 M × 24 × 1 KB = 0.96 TB, a twelfth of the whole-corpus figure. It is a disk payload keyed by video id, exactly like the segment payloads above, so it never enters the resident, replicated tiers.

Query cost

This is marginal serving cost only: the compute and I/O one extra query consumes. Index memory is a fixed cost, priced separately below.

GPU: 603 GFLOP / 60 TFLOP/s = 10.1 ms of one A10G
     one A10G serves ~99 QPS -> $0.75/hr / (99 × 3,600 s) ≈ $0.0000021/query
CPU + I/O, costed the same way             ≈ $0.0000058/query
                                              -----------
                                           ≈ $0.0000079/query
1 M queries/day  ≈  $7.90/day  ≈  $2,880/year of marginal serving

Serving is nearly free. The instinct is to optimize the per-query path, but the per-query path is $8 a day.

The index, priced

Having called the index the money, price it. Three inputs, all assumptions to challenge:

resident tiers               5.10 TB lexical + 1.16 TB vector  =  6.3 TB
replication factor           3  (one primary + two replicas: survives one
                                 machine loss, and lets a replica rebuild
                                 without a read-path outage)
                             6.3 TB × 3  =  18.9 TB resident
RAM price                    $7.00/GB-month  (a 64 GB machine at $0.62/hr;
                                              RAM is what you are buying)
18,900 GB × $7.00/GB-month   =  $132,300/month  =  $1.59 M/year

About $1.6 M a year of resident index stands against $2,880 a year of marginal serving (the index costs 550x the per-query path), plus $2.08 M once for the ASR backfill. Put the three on one line and the priority orders itself:

resident index, recurring     $1.59 M/year
ASR backfill, one-time        $2.08 M once
marginal serving, recurring   $2,880/year

A tier that is not resident is nearly free, which is why the segment payloads and the visual segment tier live on disk keyed by video id and stay out of the 18.9 TB. Moving either into RAM would be a quarter-million-dollar decision. And n-best transcript indexing is the one proposal that moves this line: it adds 4.54 TB resident, and at 3x replication and $7 per GB-month, 4.54 TB × 3 × 1,000 × $7 × 12 ≈ $1.14 M/year, nearly doubling the index bill, in exchange for 6.1 points of entity recall@100. A real trade with a real price.

Failure modes

Production finds six ways to break this system, each with the same three handles: the mechanism, the number that detects it, and the control that limits it. The summary table is at the end.

Clickbait and keyword stuffing

The mechanism is structural: two of your five channels are written by a party whose payoff is your traffic.

title        "I FIXED MY CIVIC BRAKE PADS TORQUE SPEC ROTOR CALIPER *SHOCKING*"
description  "civic brake pads, honda brake pads, ... brake torque spec ..." × 40 terms
transcript   [music] "hey guys welcome back, smash that like button" ... no torque
             value is ever spoken
OCR          (none)

BM25F on the creator-written fields:  strong
BM25F on the content-derived fields:  near zero

The defense is corroboration: trust a creator-written term only to the extent that a content-derived channel independently attests it. The evidence comes from splitting the 0.72 title posterior (which was measured over the whole corpus, mixing honest and stuffed titles) on whether the term also appears in a content-derived field:

P(about term | in title AND in transcript with k>=3)   =  0.89
P(about term | in title AND NOT in transcript/OCR)     =  0.19

An uncorroborated title term (0.19) is worth less than a densely repeated transcript term (0.58), not more, which inverts the boost you would have applied from the aggregate. Implement it as a multiplicative gate on the creator fields:

score_creator_fields  ×=  0.25 + 0.75 · corroborated_fraction

At zero corroboration the creator fields keep a quarter of their weight, a floor, not a zero, because a brand-new upload has no transcript yet. At full corroboration the multiplier is 1.0. Measured: spam-labeled results in the top 10 fall from 6.1% to 1.4%, and NDCG@10 on the healthy slice moves −0.2 points. Worth paying.

The code below implements the gate and the BM25F it modifies. The two docstrings carry the two things that make this different from a textbook BM25F.

CREATOR_FIELDS = ("title", "description", "tags")
CONTENT_FIELDS = ("transcript", "ocr")
MIN_CORROBORATING_TF = 3       # saying it aloud once must not count


def corroboration_multiplier(query_terms, field_tf):
    """Scale creator-written field contributions by how much of the query the
    content-derived channels independently attest.

    The floor of 0.25 keeps a brand-new upload -- whose transcript does not
    exist yet -- retrievable at all; that floor is also the freshness attack
    surface (see Freshness below).
    """
    if not query_terms:
        return 1.0
    hits = sum(
        1 for t in query_terms
        if any(field_tf.get(f, {}).get(t, 0) >= MIN_CORROBORATING_TF
               for f in CONTENT_FIELDS)
    )
    return 0.25 + 0.75 * (hits / len(query_terms))


def bm25f_score(query_terms, field_tf, field_weight, idf,
                field_len, avg_field_len, k1=1.2, b=0.75):
    """Field-weighted BM25F, length-normalized, gated on corroboration.

    Published BM25F does NOT saturate each field and then sum them. That
    variant is unbounded in the number of fields: put a term in one more field
    and the score rises again, with no ceiling -- exactly the keyword-stuffing
    move this section defends against. Real BM25F pools a length-normalized,
    field-weighted term frequency ACROSS fields first, then saturates the
    pooled quantity ONCE, so a term's contribution is bounded by its IDF no
    matter how many fields it is smeared across.

    Length normalization is the other half: without the 1 + b*(len/avg - 1)
    factor, tf=1 in an 8-word title and tf=1 in a 1,800-word transcript both
    score tf/(tf+k1)=0.4545, so dumping a keyword once into a long transcript
    is free. With it, the long field is discounted by how much longer it runs.
    """
    gate = corroboration_multiplier(query_terms, field_tf)
    total = 0.0
    for term in query_terms:
        pooled = 0.0                             # weighted tf pooled ACROSS fields
        for f, w in field_weight.items():
            tf = field_tf.get(f, {}).get(term, 0)
            if not tf:
                continue
            length = field_len.get(f, 1)
            avg = avg_field_len.get(f, 1) or 1
            norm_tf = tf / (1.0 + b * (length / avg - 1.0))   # per-field length norm
            pooled += w * norm_tf * (gate if f in CREATOR_FIELDS else 1.0)
        term_score = idf.get(term, 0.0) * pooled / (k1 + pooled) if pooled else 0.0
        # Saturation applied ONCE to the pooled count, so a term can never
        # contribute more than its IDF however many fields it is stuffed into.
        assert term_score <= idf.get(term, 0.0) + 1e-9, (
            "field stuffing broke the saturation bound: pool across fields, "
            "then saturate once -- never saturate per field and sum")
        total += term_score
    return total

Two subtleties, restated: saturation is applied once, to a count pooled across all fields. The per-field-then-sum variant is unbounded in field count, and the assert is its regression test. And each field’s term frequency is divided by how much longer that field runs than average, so dumping a keyword into a long transcript is not free. On top of the code, a satisfied-click quality prior in the blend stage catches videos that pass every content check and still waste the user’s time, the only signal that does.

ASR errors compounding into retrieval misses

Two ASR errors (“redeye” heard as two words, “whine” heard as “wine”) remove two of four content terms, and each arm then fails for a different reason:

query        "hellcat redeye supercharger whine"
gold video   transcript says "hellcat red eye" (split) and "supercharger wine"
lexical      "redeye" and "whine" match 0 postings -> 2 of 4 terms lost ->
             BM25F below the shard cutoff -> the video is never a candidate
dense        cosine 0.61, rank 340 of 500 -> survives retrieval, dies in L1
             because every lexical feature is zero
result       gold video not in top 100. NDCG@10 = 0 for this query.

The failure is total because it happens at retrieval. The document never becomes a candidate, so no downstream ranker can recover it. Two structural mitigations, neither needing a better ASR model: index a compound-split variant of every out-of-vocabulary token (post redeye and also red and eye), and give dense retrieval a candidate floor (reserve ~150 slots so it cannot be crowded out by a confidently-wrong lexical arm; in the trace it did find the video at rank 340, it just lost the merge).

The long tail with no good result

Fifteen percent of queries have nothing in the corpus that genuinely satisfies them. NDCG has no opinion: showing ten barely-relevant results scores better than showing nothing, so the metric rewards filling the page with junk. The reason is step 4 of the NDCG definition (divide by the best possible ordering), and “best possible” is not fixed. It depends on which pool you compare against, and there are two common conventions:

query                     "2011 subaru forester timing belt idler pulley torque"
best available result     a 2014 Forester general timing belt job, grade 1
NDCG@10, ideal from the query's own label pool   1.000  (grade 1 is all there is,
                                                        so this page IS the ideal)
NDCG@10, fixed grade-4 ideal, gain 2^rel - 1     0.067  ((2^1-1)/(2^4-1) = 1/15)
user outcome                                     3 clicks, 3 back-buttons, exit

Neither convention can say “failure.” One calls the page perfect; the other calls it a weak query on a scale with no failure state. And showing nothing scores 0 under both, worse than showing junk. The convention changes the number by 15x and changes nothing about the blindness: when a parameter choice moves the number an order of magnitude without moving the conclusion, the metric is the wrong instrument.

So carry a separate number. Zero-good-result rate is the fraction of queries where no returned result is graded >= 3, with its own launch gate. It also justifies a product surface NDCG never would: a “no strong match” state offering a broadened query, a related channel, and the closest partial answer labeled as partial. Measured, that surface raised session-level satisfaction on the affected slice by 9 points while lowering CTR, which is exactly why CTR cannot be the goal.

Freshness, and partial documents

The five channels do not arrive together. Written from the moment of upload:

t = 0 s      title, description, tags, channel      available at upload
t = 40 s     visual embeddings, thumbnail OCR       transcode + encode
t = 3 min    ASR transcript                         after transcode completes
t = 15 min   full OCR over sampled frames
t = 6 h      early engagement priors

A breaking-news video must be findable in seconds, but at t=10 s it has only the creator-written channels, the two you just learned to distrust. The freshness path is structurally the spam-vulnerable path. Resolution: index at t=0 into a small “fresh” shard with a separate scorer that leans on channel-level priors (historical corroboration rate, subscriber base, past policy strikes) instead of content corroboration it cannot have yet, and re-index into the main shard when the transcript lands. Cap the fresh shard at 2 of the top 10 slots except for queries classified as news-seeking.

Near-duplicate flooding

One popular clip gets re-uploaded 400 times. Deduplicating by content hash catches exact re-encodes and nothing else: not a crop, a mirror, or a spliced-on intro. Three detectors stacked, each line cumulative:

exact SHA of the media                                    catches 31% of pairs
+ video pHash, TMK/PDQF over keyframes                    catches 84%
+ ASR shingle, MinHash over transcript 5-grams, J >= 0.8  catches 96%

SHA is a cryptographic hash of the raw bytes, matching only byte-identical files. A pHash (perceptual hash) is a short code computed from a frame’s appearance, so visually similar frames get similar codes; TMK and PDQF are the standard video/image variants. A shingle is a sliding window of consecutive words (five here), MinHash estimates how much two sets of shingles overlap without comparing them directly, and J is Jaccard similarity (intersection over union), so J >= 0.8 means the transcripts share at least 80% of their five-word windows.

The transcript shingle is the cheap win nobody expects: a re-upload has the same words in the same order even after the video is cropped, mirrored, and sped up 2%. Cluster the near-duplicates, pick a canonical member by upload time and channel authority, and collapse the rest behind a “400 similar videos” control.

Cross-lingual mismatch

A Spanish-language repair video may be the best answer to an English query, and the searcher may or may not want it.

lexical arm     0 match — different vocabulary entirely
dense arm       matches, if the encoder is multilingual

The decision is a product one, not a model one: surface cross-language results below the fold with a language badge, and make the per-locale ratio a tuned parameter. Do not translate queries into every corpus language. That multiplies retrieval fan-out by the language count and returns worse results, because query translation loses exactly the entity terms the query depended on.

Summary

Alternatives considered and rejected

Each rejection is quantitative: a number, not a preference.

Conclusion

The design turns on a handful of load-bearing facts:

• A video is five evidence channels, and they are not equal. The transcript carries ~75x the lexical surface area of the title but far worse per-term precision, and two of the five channels are written by an adversarial creator. Field weights come from a measured posterior (via BM25F log-odds), not from intuition.
• Route, do not blend. A 4 ms classifier keeps the visual arm off 77% of traffic and the quote index off 83%, which is the only reason either is affordable.
• Retrieve on whole videos; localize at rank time. Segmenting the index destroys recall on long queries; a per-video segment payload, fetched for the 100 survivors, gives the deep link for free.
• Two-stage ranking is a cost argument. The cross-encoder is 172 ms over 1,700 candidates and 11 ms over 100, so a cheap GBDT narrows first.
• The cost is the index, not serving. ~$1.6 M/year of resident RAM and a one-time ~$2 M ASR backfill against ~$2,880/year of marginal serving. Anything kept on disk keyed by video id is nearly free.
• The right metric is never one number. WER hides entity loss, aggregate NDCG hides both the zero-good-result tail and the accent-correlated recall gap, and CTR rewards clickbait. Slice, and carry a separate gate for each failure the primary metric cannot express.
• Measure ranking changes by interleaving, not A/B. Examination cancels instead of being estimated, turning a week-long experiment into a same-day one.

One line to remember: a video is five unequal evidence channels, so every hard call in this design is really a call about which channel to trust for which query.

Further reading

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009): the source of BM25F and field-weighted scoring.
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021): the contrastive dual-encoder recipe the visual arm follows.
• van den Oord et al., Representation Learning with Contrastive Predictive Coding (2018): the InfoNCE objective.
• Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper, 2022): a representative modern ASR system.
• Malkov & Yashunin, Efficient and Robust Approximate Nearest Neighbor Search using Hierarchical Navigable Small World Graphs (2018): the HNSW index.
• Cormack, Clarke & Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (2009): RRF.
• Chapelle et al., Large-Scale Validation and Analysis of Interleaved Search Evaluation (2012): team-draft interleaving.
• Joachims et al., Unbiased Learning-to-Rank with Biased Feedback (2017): position bias and inverse-propensity correction.
• Järvelin & Kekäläinen, Cumulated Gain-Based Evaluation of IR Techniques (2002): the origin of NDCG.

Next: the content-moderation lesson.