How to design a visual search system

One photo becomes ~20 buyable listings through three stages; the load-bearing decisions are the training objective and which duplicates the index shows, not the network.

Pipeline and terms

• Three components: an encoder (image to 512 numbers), an ANN index (nearest vectors among 200M products in ms), a reranker (orders survivors). Standard two-stage retrieval: cheap candidates, then expensive scoring.
• Embedding: fixed-length vector, trained so same-thing lands close, different-thing far.
• Cosine similarity: cosine of angle between embeddings, 1.0 (same) to -1 (opposite). L2-normalize so dot product = cosine.
• SKU: retailer id for one distinct product; two sellers of one chair share a SKU.
• Recall@k: share of correct answers found in top k (the ceiling on everything downstream). Precision@k: share of top k that is correct.

Routing: threshold from cost, not a sweep

• “Similar” is four questions; the two that conflict are exact-product (invariant to color) and same-style (sensitive to color). One embedding cannot serve both, so use two heads.
• Decision “same physical item” priced in margin: C_fp = $14 (wrong “same item”, shopper returns and distrusts), C_fn = $0.60 (real match sent to style lane). Say yes when p > C_fp/(C_fp+C_fn) = 14/14.6 = 0.96.
• A cosine is not a probability: calibrate on the 3,000-query gold set. Last bucket clearing 0.96 sets tau_exact = 0.83. Result: 32% exact lane, 68% style lane. The style lane is a consequence of the threshold, not a bolted-on feature.
• Threshold belongs to an encoder version (reship the calibration table per model); insensitive to the cost estimates (a 4x swing moves it one bucket).

flowchart TD
    Q(["User photo"]) --> D{"top-1 cosine > 0.83?"}
    D -->|"yes · 32%"| E["Exact lane: rank sellers<br/>by price, availability"]
    D -->|"no · 68%"| S["Style lane: rerank by<br/>visual similarity + category prior"]
    E --> R(["Results"])
    S --> R

Objective: a relation, not a class

• Learn f(image) -> point such that for triplet (anchor, positive, negative): cos(f(a),f(p)) > cos(f(a),f(n)) + m. No class, so it survives 3% weekly catalog churn.
• Don’t reuse a classifier’s penultimate layer: softmax over 4,800 leaf categories has zero gradient for which of ~42,000 items in a leaf this is, so it collapses the exact distinctions retrieval needs (recall@10 0.31 vs 0.68 for mined-negative InfoNCE). Keep the class head only as a 0.1-weight auxiliary loss, annealed to zero.

The loss and negatives (the whole game)

• Triplet L = max(0, d(a,p) - d(a,n) + m), d = 1 - cos, has an exactly-zero-gradient region: after epoch 1, 96.4% of random triplets teach nothing (~28x the gradient variance you think you have).
• InfoNCE with temperature tau=0.07: one mined hard negative carries ~7,300x the gradient of a random one. A 16x bigger batch buys only ~14x; mine first.
• The trap: the hardest negative is a mislabeled positive 16% of the time (same product, different seller/photo), carrying the largest weight. Fix with a semi-hard band (sample below the top-1, e.g. cos in [0.55, 0.70]) and dedupe the catalog first (perceptual hash, cluster above 0.97 cosine).

Encoder, index, training

• Encoder: ViT-B/16 at 224px, 86M params, 34.9 GFLOP, two 768->512 heads, L2-normalized, fp16. Only the exact head is indexed; the style head reranks the same candidates. Pixels only (must be knowable from a fresh photo). Version-pinned: a v2 query can never reach a v1 index.
• Index: HNSW (continuous inserts beat IVF at 3% churn), ~1,172 B/vector = 234 GB. efSearch trades time for recall at zero memory; at M=16, efSearch=128 the graph keeps 0.98 of true top-100 (costs 1.4 of the 29.7 missing points; encoder + query distribution cost the rest).
• Tombstones: HNSW has no true delete; dead vectors accumulate at 3%/week. Compaction (rebuild live-only) is the only reclaim, and its interval is the takedown SLA. Binary quantization + rescore: 5.5x smaller payload, 0.96 vs 0.98 recall, ~3.4x cost cut.
• Labels are free (product ids, purchases, returns); annotation buys only the 3,000-query gold set. Co-view is a category signal (P(same product)=0.34), right for style, wrong for exact.
• Training gotchas: split on time and content (group on content-cluster id; point-in-time join on 30-day counters or leak 13 NDCG points). logQ subtracts log sampling frequency, worth ~0.48 of cosine (6x the router’s band) so one global tau is legal. Retrain biweekly (economic-order-quantity optimum ~2.1 weeks; curve flat 1-4 weeks). Cold-start (random-seed) re-encode quarterly: v1/v2 cosine of the same image is only 0.31.

Metrics and failure modes

• Offline: recall@k (retrieval), NDCG@10 (ranking), precision@1 at tau (exact lane), distinct content clusters in top 20 (dedupe guard). Provenance trap: a catalog-photo gold set overstates the shipped system by 21 points (0.912 vs 0.703). Slice every report by query provenance.
• Online: attributed purchase within 7 days per search. Not CTR (near-duplicate flooding raises CTR@1 while wrecking the session).