How to design video search and ranking

Text-query video search returns 10 ranked videos with jump-to timestamps; the core idea is that a video is five unequal evidence channels, so every decision is about which channel to trust for which query.

The five evidence channels

• Title, description, tags: creator-written, adversarial incentive (paid in your traffic).
• Transcript (via ASR, speech-to-text), on-screen text (via OCR), visual track (pixels): content-derived, outside creator control.
• Channels differ in accuracy, index lag (time from upload to searchable), and authorship.

• Title = high precision, no coverage; transcript = full coverage, terrible per-occurrence precision. Cannot pick one field or treat them alike.
• Transcript carries ~75x the title’s unique terms and answers 55% of queries; visual answers another 14%.

Scoring: BM25F from log-odds

• BM25F: fielded lexical scorer; per-field weights are the log-odds log(p/(1-p)) from the posterior table, so weights are measured, not tuned.
• log-odds: title +0.94, transcript(k>=5) +0.32, transcript(k=1) -2.31 (a word seen once in 1,800 argues against relevance).
• Boost ratio = exp of log-odds difference: title over dense transcript term ≈ 1.9x (not 10x).
• Needs TF saturation (5th mention worth far less than 1st) + length normalization (pool weighted TF across fields, saturate once, or field-stuffing is unbounded).

Query routing (6 classes, 5,000 labeled)

• A 4 ms linear classifier (char 3–5-grams) fires only needed arms. Visual arm runs on 23% (14%+9%), quote index on 17% (8%+9%).

• Fuse arms with RRF (1/(rank+const), ranks only, no score calibration across incomparable scales).

Visual arm & temporal granularity

• Dual encoder (CLIP recipe): ViT-B/16 vision tower (512-d/frame, offline) + text tower (512-d query); match by cosine.
• Trained contrastively (InfoNCE, temperature ~0.07) on click-mined (frame, query) pairs; shared space is the whole mechanism, so towers must train together.
• Visual arm lifts visual-descriptive recall@10 from 0.38 → 0.67 (71% of those videos name the answer in no text).
• Do NOT index 30-s segments: posting count barely changes (~1.35x), but conjunctive match breaks on long queries (recall@100 4+ terms 0.77 → 0.41). Retrieve on whole videos; localize at rank time by fetching a per-video segment payload (1.05 TB, keyed by video id, never searched) for the 100 survivors.

Serving path & latency (150 ms budget)

Query understanding 8ms → 4 arms parallel (lexical BM25F 25ms hides dense 12, visual 9)
→ merge/dedupe 3ms (~1,700) → L1 GBDT 15ms (→100) → segment fetch 18ms
→ L2 cross-encoder 11ms (→scores) → blend 4ms → 10 results

• Serial path p50 = 84 ms, p95 = 140 ms (66 ms headroom absorbs tail). p95 200 ms overall (150 search + 50 render).
• L1: GBDT, 190 cheap in-memory features, 1,700→100. L2: cross-encoder (query+doc read together), 100 pairs, judged by NDCG@10 (ship gate). Both trained on satisfied clicks, position-bias corrected.
• L2 cost: 603 GFLOP / 60 TFLOP/s A10G ≈ 10.1 ms. Over all 1,700 it would be ~172 ms — the two-stage cost argument.
• Segment fetch is I/O-bound; L2 is GPU-bound. Different levers.

ASR: highest-value, uneven failures

• Entity survival = 1 - WER_term^k (assumes independent errors across occurrences). Repetition rescues common terms, abandons rare ones said once.
• WER 0.07 at k=9 → loss ~0; WER 0.41 at k=1 → loss 0.41 (coin flip). Failure is total (video never retrieved), not graded.
• Right metric: entity WER at video level = 0.37 (not aggregate WER).
• Fixes without retraining: (a) contextual biasing (feed title/tags as shallow-fusion bias, -38% rel entity WER, ~2% throughput), (b) n-best indexing (top-3 hypotheses, postings 2.4x, +6.1 recall pts), (c) phonetic double-metaphone fallback on zero-result path.
• Accent gap: WER 0.081 (US) → 0.192 (non-native), 2.4x; amplified at k=1 (entity loss 0.26 → 0.51). Report recall sliced by accent — aggregate NDCG moves ~0.3 pts and hides it.

Metrics & experiments

• NDCG@10 (grade 0–4, discounted by position, normalized to ideal): ship gate but rewards filling 10 slots when nothing is relevant. Carry zero-good-result rate (15% of queries have no good answer), recall@1000 (retrieval), MRR (navigational), localization accuracy separately.
• Satisfied click = click AND (watch >=30 s OR >=30% duration) AND no return-to-results within 60 s. Primary online metric; reformulation rate is sharpest negative. CTR is misleading (clickbait raises it).
• Position bias: P(click) = examination(j) × relevance(r); rank 1 examined 0.72, rank 10 only 0.13 (5.5x).
• Team-draft interleaving cancels examination instead of estimating it: a 0.1-pt decision needs ~6–15k sessions vs ~3.9M queries/arm for A/B (260–650x). Unit of inference = sessions, never clicks. Interleave to shortlist, A/B the finalists.

Cost (the index is the money)

• Resident index ~$1.59 M/yr (6.3 TB × 3 replicas × $7/GB-month RAM). ASR backfill ~$2.08 M once (100 M audio-hrs / 120x batched × $2.50/GPU-hr; unbatched 20x ≈ 6x more). Marginal serving $2,880/yr ($8/day).
• Anything on disk keyed by video id (segment payloads, top-40M visual segment tier) is nearly free.
• Frame-level visual index = 180 TB (360x); video-level pooled = 500 GB and answers 91% of visual queries.

Failure modes (mechanism → detection → control)