How to design a visual search system

A shopper photographs a chair they saw in a cafe. The system shows them chairs they can buy.

In this lesson, we’ll design that system end to end, from the photograph to the purchase. The hard part is not the network architecture. It is the training objective, and in particular which wrong answers the model sees while it learns. By the end you’ll be able to size each component from the workload, name the number that forces each design move, and defend the whole pipeline in an interview.

The input is one photograph: a JPEG from a phone, a couple of megabytes, often poorly lit and often holding several objects when the shopper cares about one. The output is an ordered slate of about 20 buyable listings, with the top slot reporting one of two outcomes: the exact item photographed (with every seller who carries it), or no exact match followed by the closest items in style.

Between the two ends sit three components:

1. a model that turns a picture into a list of 512 numbers,
2. an index that finds the nearest such lists among 200 million products in a few milliseconds,
3. a ranker that puts the survivors in order.

A few terms, defined once:

• An embedding is a fixed-length list of numbers a model produces from an input, here 512 numbers from one image. It is trained so pictures of the same thing land close together and pictures of different things land far apart.
• Cosine similarity measures “close”: the cosine of the angle between two embeddings, from 1.0 (same direction) through 0 (unrelated) to -1 (opposite).
• A SKU (stock keeping unit) is the retailer’s identifier for one distinct physical product. Two sellers offering the same chair share one SKU.
• An encoder is the model that produces the embedding.
• Recall@k is the fraction of the genuinely correct answers that appear anywhere in the top k results. It measures whether the right item was found at all, before the question of ranking.

The system follows the standard two-stage retrieval pattern: retrieve a few hundred cheap candidates, then score them expensively. A triplet loss trains on three images at a time (an anchor, a matching item, a mismatched item). An approximate nearest neighbour (ANN) index trades a little exactness for the ability to search hundreds of millions of vectors without comparing against all of them. Both are developed below.

Two ideas from elsewhere, restated so this page stands alone:

• A convolutional network reuses one small set of weights at every position in an image, which lets it learn “an edge is an edge wherever it appears” from a reasonable amount of data (neural layers and architectures).
• A vector index is a data structure that, given one query vector, returns the nearest few among hundreds of millions without scanning them all, by keeping a navigable graph or a coarse partition over the vectors (vector index internals).

Framing: “similar” is four different questions

Before any modelling, decide what the system is being asked for. The word “similar” hides four different products with different labels, different metrics, and in two cases different models.

Instance-level metric learning means training a model to place two pictures of the same individual object close together, not two pictures of the same kind of object: “this chair” versus “a chair.”

The four metrics: Recall@k is the share of correct answers found in the top k; Precision@k is the share of the top k that are correct; NDCG (normalized discounted cumulative gain) scores an ordering when some answers are better than others, discounting positions further down the page; attach rate is the share of sessions where the shopper adds the suggested companion item.

The two products that get conflated are exact-product and same-style, and they want opposite invariances. Exact-product must be invariant to color shift from bad white balance; same-style must be sensitive to color, because a navy sofa and a mustard sofa are different products a shopper is choosing between. One embedding cannot serve both.

A shopping product needs exact product first, with style as the fallback when no exact match clears a threshold, which the routing logic captures directly:

flowchart TD
    Q(["User photo"]) --> D{"Is the top-1 cosine<br/>above tau_exact = 0.83?"}
    D -->|"yes · 32% of traffic"| E["Exact-product lane<br/>rank by price, availability,<br/>seller quality"]
    D -->|"no · 68% of traffic"| S["Style lane<br/>rank by visual similarity<br/>+ category prior"]
    E --> R(["Results"])
    S --> R

Every incoming photo is asked one question: is the top-1 cosine above tau_exact = 0.83? Above it, the matching item is already decided and the only job left is ranking the sellers who carry it. Below it, the system ranks by visual similarity plus a prior over which category the shopper is probably in. The threshold is the only free number in the diagram, and the two-lane architecture follows from it, so derive it first.

Where the threshold comes from

The decision is “declare this candidate the same physical item,” and a yes/no decision has two ways to be wrong. A false positive (C_fp) says yes when the answer is no; a false negative (C_fn) says no when the answer is yes. Price both in dollars of margin:

• C_fp (a wrong “same item”): the shopper buys, the box holds a different chair, they return it and stop trusting the feature. About $14 of margin.
• C_fn (a real match sent to the style lane): the shopper still sees the item, one row lower, in a slate that converts slightly worse. About $0.60 of margin.

If you are p sure the pair matches, saying yes costs (1-p)·C_fp on average and saying no costs p·C_fn. Saying yes wins exactly when p > C_fp / (C_fp + C_fn) = 14 / 14.6 = 0.96. So the probability threshold is just the share of the total error cost the false positive owns, and at 23:1 that is 0.96. The system may not say “same item” unless it is 96% sure, and everything it refuses on still has to be shown to somebody. That is the style lane: it is a consequence of the threshold, not a second feature bolted on.

A cosine is not a probability, so the shipped number comes from a calibration: an empirical table binned on the 3,000-query gold set (real query photos with exhaustively verified answers, built in the data section), saying what fraction of pairs in each cosine range really were the same product. Read down and stop at the last row still above 0.96:

So tau_exact = 0.83. The three clearing buckets hold 214 + 331 + 408 = 953 of the 3,000 queries, so 32% of traffic takes the exact-product lane and 68% takes the style lane. The fallback is the common case.

Two properties of that number:

• It belongs to an encoder version, not to the problem. The cosine-to-probability mapping is a property of the embedding, so every model version reships its own calibration table. Version pinning has to cover the threshold along with the vectors.
• It is insensitive to the cost estimates. A 4x swing in C_fp moves the threshold by only one cosine bucket in either direction. You do not need the margins to three digits, only that the ratio is tens, not ones.

The rest of the framing is four questions whose answers each change something structural:

Terms in that table: HNSW (hierarchical navigable small world) is a vector index built as a graph you walk downhill toward the query; it accepts new vectors one at a time. IVF (inverted file) partitions vectors into clusters up front and searches a few; it is cheaper to delete from and clumsier to insert into. A tombstone is a deleted item’s vector left in place and filtered out of results, because the graph still needs it as a stepping stone. p99 is the latency only one request in a hundred exceeds. CTR is click-through rate.

This framing holds only if: the catalog carries a reliable product identifier (so “same product” is a fact, not a judgement); a meaningful fraction of queries genuinely have an exact match (32% here earns the two lanes); a wrong “same item” really costs tens of times what a miss costs; and traffic is dominated by phone photos. Drop the second and you ship one ranked list; drop the third and the threshold collapses toward 0.5 and the lanes merge.

The ML objective: a relation, not a class

The input is one image and the output is 512 numbers. You are not learning a label; you are learning a function f from an image to a point in d-dimensional space such that, for every training triplet of an anchor a, a matching positive p, and a mismatched negative n:

cos(f(a), f(p))  >  cos(f(a), f(n)) + m

The anchor must sit closer to the positive than to the negative by at least a margin m. The margin keeps the constraint satisfied with room to spare instead of balanced on the boundary.

Nothing in that objective is a class, a category, or a fixed answer set. A classifier has to know its answer set in advance, and this catalog turns over 3% of its items every week, so the only thing that survives is a model that learns a relation between two images.

Why not train a classifier and reuse the penultimate layer

A common shortcut is to train an ordinary category classifier and reuse the layer before its output as the embedding. It underperforms, and the reason is worth knowing.

Such a classifier uses a softmax over 4,800 leaf categories (the most specific taxonomy level, where “lounge chairs” stops splitting). A softmax reaches its lowest loss the moment the representation is sufficient for category and nothing more. Which of the 200M / 4,800 ≈ 42,000 items in a leaf this one is carries zero gradient, so training has no reason to preserve it. The classification objective actively encourages collapsing the exact distinctions retrieval is built on.

Measured on the same backbone, data, and compute, so the only variable is the objective:

A linear probe (last column) freezes the embedding, fits the simplest classifier on top, and measures how much of a property the embedding already contains. InfoNCE is the contrastive loss derived below; read it for now as the softmax version of the triplet loss. The classifier wins the classification metric and loses the metric you ship.

The probe is top-1 accuracy on a 1,000-way SKU discrimination set (chance 0.001), not R^2. SKU id is nominal: its values are names, not quantities, so there is no variance to explain and R^2 would only reflect whichever integers the catalog assigned. (The red-dress failure below makes this concrete.)

There is one legitimate use for the classification head: as an auxiliary loss at a small weight, because category structure regularizes early training. Weight it at 0.1 and anneal to zero.

What the encoder actually is

The objective says nothing about architecture. Four decisions:

Terms: the backbone does the seeing; a head is a small layer on top that shapes the output for a task. ViT-B/16 is a Vision Transformer, “Base” size, that cuts the image into 16x16 patches and lets every patch attend to every other from the first layer. ResNet-50 is the convolutional alternative, where each layer sees only a small neighbourhood. GFLOP is a billion floating-point operations. L2-normalized rescales every vector to length 1, which makes a dot product equal to a cosine. fp16 is 16-bit floating point, two bytes per number. EXIF-oriented rotates the photo per the camera’s orientation tag; without it a quarter of phone photos arrive sideways.

Two heads is not two indexes. Only the exact-product head is indexed (200M vectors). The style head is a reranking feature: a color-invariant embedding still retrieves the right shelf, so the style lane reranks the same candidates the exact head returned, reading style vectors from the metadata store (200M × 512 × 2 B ≈ 205 GB of NVMe, about $16/month). A second ANN index would double the expensive resource (RAM) and add nothing the candidate set does not already contain. The extra head is 768 × 512 ≈ 0.46% of the backbone: the cheapest thing in the design, and what makes the two lanes expressible at all.

The encoder in summary: a ViT-B/16 with two 768->512 heads, fed one 224x224 crop (pixels only, no catalog metadata, because anything the item side knows must be knowable from a fresh photo). Labels are pairs harvested for free from the catalog’s product identifiers, purchases, and returns, never from a human annotator except on the evaluation set. It is served one forward pass per query (batch 1, 8 ms), version-pinned so a v2 query can never reach a v1 index; the item side runs offline in batches of 256.

This works only if the catalog’s photos are honest depictions (the label supply rests on “same SKU implies same object”), visual appearance is sufficient to identify the product (it is not for items distinguished only by size or fabric weight, where the reranker’s attributes must decide), and shoppers photograph one dominant object (violated on 21% of traffic, as the query-side failures show).

The loss, and why random negatives stop teaching

One thing separates a visual search system that works from one that does not: the choice of negatives. Three claims, each argued below:

1. A randomly chosen wrong answer teaches the model almost nothing.
2. One deliberately chosen near-miss teaches it thousands of times more.
3. The strategy that mines the hardest near-misses walks into a trap you have to design around.

A batch is the group of examples seen before one weight update (256 or 1,024 images here); an epoch is one full pass over the training set; a negative is a wrong answer shown on purpose, and pushing it away is what teaches the model what “right” means.

Triplet loss has an exactly-zero-gradient region

The triplet loss charges you only when the objective is violated:

L = max(0, d(a, p) - d(a, n) + m)          d = 1 - cos

If the negative is already comfortably further away than the positive, the loss is 0 and the gradient is exactly zero: the example changes no weight. How often is a random negative already that easy? With 200M items in 4,800 leaf categories, a random item lands in the anchor’s category with probability 1/4,800 ≈ 2.1e-4, so an in-batch scheme at batch 1,024 gives each anchor only 1,024 × 2.1e-4 ≈ 0.2 same-category negatives. An in-batch scheme reuses the other items already in the batch as negatives, for free. Four out of five anchors see no informative negative at all.

Empirically, after the first epoch 96.4% of random triplets have zero loss, so only 3.6% of the batch contributes gradient. That is worse than it sounds, because gradient variance scales as 1/n_effective: an effective batch of 3.6% means roughly 28x the gradient variance you think you have. The wall-clock cost per useful gradient is 28x the number on your dashboard.

The InfoNCE version, where the vanishing is quantitative

InfoNCE (information noise-contrastive estimation) is a softmax contrastive loss: instead of one negative at a time, it scores the positive against N negatives at once and asks the model to pick the positive from the lineup. A temperature tau divides every score before the softmax, so a small tau makes the loss care almost only about the closest competitors. The gradient weight on each negative equals its share of the softmax denominator, so a negative whose share rounds to zero is one the model never learns from.

Take tau = 0.07, the positive at cosine 0.80, and 255 random negatives at cosine 0.10 (what random items from a 200M catalog look like). The positive’s exponential term dominates the denominator: it owns 98.9% of it, the loss is a tiny 0.0115, and each random negative carries a gradient weight of about 4.5e-5. That is the vanishing, in numbers.

Now add one mined hard negative (a different chair that looks nearly identical) at cosine 0.75. Because the exponential turns a 0.05 gap in cosine into a factor of exp(0.05/0.07) ≈ 2, that single negative grabs a third of the denominator: the loss jumps 35x to 0.406, and the negative carries a weight of 0.33. Compared side by side, one item at cosine 0.75 against 255 at cosine 0.10:

• loss: 0.0115 -> 0.406 (35x)
• per-negative weight: 4.5e-5 -> 0.33 (about 7,300x)

So one hard negative carries roughly 7,300 times the gradient of one random negative. That is the whole argument for hard-negative mining.

The corollary: scaling the batch is an expensive way to buy what mining buys for free. Going from 256 to 4,096 in-batch negatives multiplies the total negative-side gradient by only about 14x (sublinear, because the positive’s softmax share is already near 1 and can only fall so far), while costing 16x the memory. One mined negative moved the same quantity by 29x. A 16x batch buys about half of what a single mined negative buys.

Mining strategies, and the trap in the best one

There are five ways to choose negatives, trading cost against quality:

A memory bank is a rolling cache of embeddings from earlier batches, letting you draw negatives from far more items than fit in one batch; the price is that those cached vectors drift out of date. Offline ANN mining builds a search index over the current embeddings between epochs and asks, for each anchor, which non-matching items it currently ranks closest, a direct search for the model’s own worst confusions.

The trap: the hardest negative is very often an unlabeled positive, a genuine match your labels missed. On a real catalog, mining the top-1 nearest non-labeled item per anchor turns up, on a hand-audit of 500:

genuinely different product        82%
same product, different seller     11%   <- false negative
same product, different photo       5%   <- false negative
same product, different colorway    2%   <- ambiguous

The 16% (11% + 5%) of false negatives carry the largest weight in the batch (the 0.33 above), so you would be training the model, with maximum force, to separate items you actually want together. A colorway is one color variant of an otherwise identical design; whether it counts as the same product is exactly the exact-product-versus-style question, which is why that row is ambiguous.

Two cheap fixes:

• Semi-hard band. Instead of the single hardest negative, sample from a band whose upper edge sits strictly below the mined top-1, e.g. cos(a, n) in [0.55, 0.70] when the positive is at 0.80. This excludes the 0.75 region where false negatives concentrate. A negative at cosine 0.60 is still worth about exp((0.60-0.10)/0.07) ≈ 1,300 random ones, so you give up a factor of ~6 against the riskiest pick and remove most of the hazard.
• Dedupe before mining. A perceptual hash is a short fingerprint that is identical for visually near-identical images, turning duplicate-finding into a lookup. Run one over the catalog with an exact-embedding pass, cluster items above 0.97 cosine, and treat a whole cluster as one entity for mining and evaluation. That cluster id is reused for the train/test split, slate dedupe, and the diversity guardrail, so it is worth building once.

Data: labels without a single annotation

Every “these two images are the same product” label already exists inside the business. Six sources, not interchangeable. P(same product) is the chance a pair from that source really is the same physical item; P(same category) the chance it is merely the same kind of thing:

The co-view row is the one to watch, because it is a category signal, not a product signal. At P(same product) = 0.34, two thirds of co-view “positives” are wrong for an instance-level objective, and the model learns to pull together items that merely share an aisle. That is the right label for the style lane and the wrong one for exact product.

An augmentation is a deliberately distorted copy of an image (cropped, recolored, blurred, re-compressed) paired with its original. It manufactures a guaranteed positive, which is why P(same product) is exactly 1.00 and why it teaches nothing except the distortions you chose.

The first three rows plus the mined negatives stack into a curriculum, easiest signal first:

stage 1   augmentation pairs only        learns cheap invariances, converges fast
stage 2   + same-SKU multi-photo         the main signal, ~1.7B pairs
stage 3   + user-photo/purchase pairs     closes the domain gap to real queries
stage 4   + mined hard negatives          the largest single jump

The augmentation set is where you install invariances deliberately, and also where you install the wrong ones. Never color-jitter and the model is free to use color as its primary discriminator (the red-dress failure). Jitter aggressively and exact-product matching breaks for products that differ only in colorway. The augmentation policy is a product decision, not a pipeline setting.

Human labeling appears in exactly one place: the evaluation set. It is 3,000 real user query photos, each with an exhaustively verified list of matching SKUs, stratified by category and photo quality. At 8 minutes each, that is 400 hours, about $8,800 (infrastructure, not a project). Without exhaustive verification, recall@k is uncomputable, because you cannot divide by a denominator you do not have. This is also the set the routing calibration is binned on, so its cost buys the threshold as well as the recall number.

This data plan holds only if the product-identifier join is largely correct (a SKU split across two ids becomes a false negative in mining and a false miss in evaluation), purchases can be attributed back to the search within 7 days (what makes user photos labelable), and the 4.6 photos per SKU are not all concentrated in popular items (or stage 2 teaches only about the head).

Training: the split, the correction, and the cadence

Turning the loss and the pairs into a run has four decisions, each a place this system will quietly cheat itself if you take the default.

Split on time and on content, because there are two leaks. A leak is any path by which test-set information reaches the model during training, inflating the score. Train on weeks 1-8, validate on week 9, test on week 10, gold set frozen out of all three.

leak 1   a SKU with 9 catalog photos yields 36 same-SKU pairs. A random split
         puts (A,B) in train and (A,C) in test, so the test pair shares an image
         with a training pair and measures memorization.
         -> group the split on the content-cluster id (from mining dedupe).

leak 2   the reranker's ctr_30d / purchase_rate_30d / return_rate are counters.
         Computed today for a training row from six weeks ago, the 30-day window
         contains the click that IS that row's label.
         -> point-in-time join on the impression timestamp.

A point-in-time join computes every feature as it stood at the moment the row was logged, not as it stands today (temporal features and lookahead leakage). Skipping the rebuild takes offline NDCG@10 from 0.71 to 0.58, and the 13 points that vanish were never real. A team that skips it ships a reranker that measures better and ranks worse.

logQ, and why this chapter cannot skip it. In-batch negatives come from impressions, so a popular item appears as somebody’s negative in proportion to its popularity, and every such appearance pushes its score down. The fix subtracts, from each item’s score, the log of how often it gets sampled:

logit'(a, i)  =  cos(f(a), f(i)) / tau  -  log Q(i)

Q(i) = P(item i appears as an in-batch negative), a decayed streaming count
       per content-cluster id

Head-to-tail impression share spans about 1,000x here, so the correction spans log(1,000) = 6.9 nats of logit, and at temperature 0.07 that is 6.9 × 0.07 ≈ 0.48 of cosine. That is six times the entire 0.83-to-0.75 band the router lives in. Uncorrected, a cosine of 0.83 means “same item” for a tail chair and “not remotely” for a head chair, and a single global tau_exact is not a legal object; the threshold would have to be a 200M-row table. logQ is what buys the one number the architecture is built on.

What logQ does not buy matters as much. It fixes the score scale along the popularity axis. It does nothing for a tail item that appears in too few positive pairs to have learned an embedding at all; that is a data problem, fixed on the positive side by upsampling tail SKUs’ multi-photo pairs in stage 2. Two failures, two controls, and conflating them is the common error.

Cadence is derived, and the thing that decays is not index freshness. A new listing is retrievable minutes after its photo is encoded, so freshness is an index property. The encoder decays only as the catalog’s visual distribution drifts. Freeze one model and keep scoring it against fresher gold data: recall@100 on the low-light user-photo slice goes 0.752 -> 0.721 over 26 weeks, a decay of about 0.0012 recall per week.

Turning a decay rate into an interval is an economic-order-quantity balance: waiting k weeks means the model is stale by an average of r·k/2 recall, while retraining every k weeks amortizes the deploy cost Y over those weeks. The optimum is k* = sqrt(2Y / (V·r)), where V is the value of a unit of recall per week and Y the cost of one retrain-and-deploy. With a point of recall worth about $4,200/week (600k searches/day, 2.1% purchase rate, $12 margin, and a measured 0.4% relative purchase lift per recall point) and Y ≈ $1,136 (a $297 training run, $16 re-encode, $223 dual-index memory window, half an engineer-day), k* ≈ 2.1 weeks:

Biweekly, and the curve is flat: every interval from 1 to 4 weeks lands within 30% of the optimum. Monthly costs about $960/month more but halves the number of atomic index swaps, and since an index swap is the operation behind the worst failure mode (embedding drift), trading a little money for half as many exposures to it is a real argument. There is also a floor the economics cannot see: the user-photo/purchase pairs need the 7-day attribution window to close, so no cadence below one week exists regardless of k*.

Warm start does not buy you out of the re-encode. A warm start begins the new run from the previous model’s weights instead of random, converging 5-10x faster. The temptation is to conclude the new space is close enough to skip the 200M-image backfill. It is not: cosine between v1 and v2 embeddings of the same image is only 0.31, and warm-starting narrows that gap without closing it, and “narrower” is not a property the index can use. Cold-start once a quarter as a drift control, because a warm chain makes the model a function of its own history and hides the degradation the frozen probe set exists to catch.

The bill. A training step is a forward plus a backward pass, about 3 × 34.9 ≈ 105 GFLOP per image encoded. Across the four curriculum stages that is roughly 1.1 billion encodes, about 107 GPU-hours, plus 12 for the ANN mining rebuilds: $297 total at $2.50/GPU-hour. A full reindex is $16. So neither compute nor money is the constraint on shipping a new encoder. The constraint is the atomic swap and the 2x memory window it needs, which is why the cadence table is priced on the deployment, not the training. That plan assumes the visual distribution drifts slowly, the popularity distribution is stationary enough for a streaming count to estimate Q, and you can hold two full indexes in memory during a deploy.

Retrieval and ranking

The serving half is an index that finds a few hundred candidates out of 200 million, a reranker that reorders them using information the index cannot see, and a latency budget that decides whether any of it ships.

The index

The index is the most expensive object in the system, and its cost is memory, not throughput. It holds 200M items at d = 512 in fp16, one embedding each after per-product pooling. An HNSW graph costs about 1,172 bytes per vector: 1,024 for the fp16 payload, ~128 for the layer-0 neighbour links, and a few bytes for the sparse upper layers and bookkeeping. The payload is 87% of the bill, because the graph structure above layer 0 is nearly free. That is 200M × 1,172 B ≈ 234 GB.

The reason to shard is memory, not QPS (queries per second). That changes the failure mode: losing a shard costs recall on a slice of the catalog, not availability, so the system degrades quietly. Monitor per-shard hit counts.

The search-time dial is efSearch (how many candidate nodes the graph walk keeps alive); it trades time for accuracy at zero memory cost. At M = 16, efSearch = 128 the graph returns 0.98 of the true top-100. Splitting the end-to-end recall into what the encoder and query distribution can achieve against a perfect index (0.717) versus what the approximate graph keeps of it (0.98):

0.98 × 0.717 = 0.703 end to end
of the 0.297 missing from a perfect 1.0:  the graph costs 1.4 points, the encoder
and query distribution cost the other 28.3.

A team that spends a quarter tuning efSearch is optimizing the 1.4.

The delete problem

HNSW was chosen over IVF because 3% weekly turnover means continuous inserts. The other side of that choice is that HNSW has no true delete: a delisted item is tombstoned, filtered from results but still present as a routing hop. At 3% per week, tombstones accumulate fast:

tombstones after w weeks = 200M × 0.03 × w   (live stays at 200M)

w = 13   78M dead   28% of the index   1.39x memory
w = 52  312M dead   61%                2.56x   -> 39 machines

The four-shard layout has only about three weeks of headroom on 64 GB machines, so the space has to be reclaimed. Compaction rebuilds a shard from scratch containing only its live vectors, the only way a graph index reclaims tombstone space. Rebuild one shard at a time behind an alias (a name pointing at the current index, so readers move between versions without knowing). One 40M-vector shard rebuilds in about 25 minutes on 32 threads for under a dollar.

The shipped layout is five shards, compacted one every 5 days, giving each a 25-day cycle whose peak tombstone load (about 11%) still fits 64 GB. Four shards on the same rotation peak at 63.6 GB against 64, which is a coincidence, not headroom. Five shards replicated 3x is 15 machines. The compaction interval is also a commitment, not a preference, because a delisting is sometimes a takedown (counterfeit, recalled, or an erasure request), and a tombstoned vector still exists in RAM and every snapshot. The compaction interval is the takedown SLA.

The compression lever

Since the index is almost all of the bill, one compression is worth pricing. Binary quantization stores each dimension as a single bit (is it positive?) instead of two bytes, then rescores the few hundred survivors against their true full-precision vectors. It compresses the payload only, because the links are incompressible integers, so per-vector cost falls from 1,172 B to 212 B (a 5.5x win, not 8x). That is 42 GB, one shard on three machines instead of fifteen.

The term people drop is that rescoring needs the full vectors somewhere: 200M × 1,024 B ≈ 205 GB of fp16 on local NVMe (solid-state disk, slower than RAM, far cheaper per gigabyte), about $16/month plus ~2 ms of random reads. And recall is not free: binary is 0.96 rescored against 0.98 for fp16. Two points of index recall doubles the index’s own contribution to the error budget (1.4 -> 2.9 points), even though its share of the total shortfall barely moves. It is still a good trade (a 3.4x cost cut, priced below), and it should be stated that way instead of sold as free. It fits here specifically because the exact-product lane rescores its top 200 anyway before comparing against tau_exact.

The reranker

The system needs a second model for one reason: the ANN score is a single dot product between two vectors computed without ever seeing each other, and much of what decides a purchase is a property of the pair. That independence is what makes indexing possible (you can precompute an item’s vector because it does not depend on the query) and is exactly what the reranker exists to undo:

Earth-mover distance between two color histograms measures how much color mass you would move to turn one palette into the other, more forgiving than exact matching. OCR is optical character recognition, reading text baked into pixels (a logo) to compare against the brand field.

The model is a GBDT (gradient-boosted decision tree ensemble), the standard winner on tabular features (ensembles and boosting). Over ~60 features it costs about 3 microseconds per item, so scoring 200 candidates is 0.6 ms of trees against 8 ms of feature assembly: fetching the numbers costs thirteen times more than using them. It is trained on logged slates (clicks, purchases, returns attributed back to the search, graded exact = 3 / same style = 1 / other = 0) with the point-in-time join and temporal split above. Its ceiling is retrieval’s recall (a reranker cannot retrieve), it needs catalog attributes to be populated, and it needs engagement counters that are null for new listings, which is the cold-start failure below.

The latency budget

The encoder’s cost you can derive. A ViT-B/16 at 224px makes (224/16)^2 = 196 patches plus one CLS token (a learned slot whose final value represents the whole image), 197 tokens over 12 layers at width 768. The matrix multiplies dominate at 33.5 GFLOP; the all-to-all attention, quadratic in 197 tokens, adds only 1.4, for 34.9 GFLOP/image. Every stage then gets a line, at p50 (median) and p99:

                                                  p50    p99
upload + JPEG decode + resize                      45    108
safety / moderation classifier                      6
object detection + crop                            18
ViT-B/16 encode, batch 1                            8           launch-bound at
ANN, 5 shards in parallel, efSearch 128             3     25    batch 1, not
eligibility filter + metadata fetch, 800 cands      6           FLOP-bound
rerank 200 candidates (assembly 8 + score 0.6)      9
content-cluster dedupe + MMR diversity              2
jitter across stages with no p99 column                  +8
                                                  ---    ---
                                                   97    190

“Launch-bound at batch 1” means the GPU spends more time being told what to do than doing it, because one 34.9 GFLOP image does not fill it. MMR is maximal marginal relevance, the diversity step below. The index is asked for 800 candidates but the reranker scores only 200, because near-duplicate collapse and filtering eat the difference.

190 ms p99 against the 150 ms requirement misses by 40 ms; this configuration is not shippable. A p99 is dominated by whichever single term has the fattest tail, not by the sum, and here that term is upload on a mobile network (45 -> 108 ms). Downscaling the image client-side to 640px before upload takes upload’s p99 to 30 ms, and the end-to-end p99 to about 112 ms, which clears 150. That subtraction is only legal because one term dominated; once upload stops dominating, the p99 is set by several medium tails at once and has to be measured. Hedging the ANN fan-out (re-issuing to a second replica for any shard silent at 8 ms) is margin, not another clean subtraction.

Two consequences: half the p50 and the whole p99 problem are upload and decode, which is not a model problem, so the levers are a client change and streaming the first results before the reranker finishes. And the client downscale is a launch dependency, not an optimization: the server-side downscale in the serving diagram is only a fallback for old app versions, and it runs after the 45 ms has already been spent on the wire.

Metrics

Three questions: what to measure offline per stage, why the offline number is about 21 points too optimistic if you build the eval set the obvious way, and which online metric to ship against.

Offline

Each stage gets its own metric, because a stage can only be held responsible for its own job:

For the lane, quote the cumulative precision across everything the lane admits (0.976, the query-weighted average of the three clearing buckets), not the marginal 0.962 of the bucket sitting on the threshold. The marginal number answers where the threshold goes; the cumulative one answers how the lane performs, and quoting the first as the second understates a shipped lane by 1.4 points. The number that means something in production is the same measurement on held-out traffic; a drop there is the threshold going stale, not the ranker.

The evaluation error that dominates this problem: your gold set is catalog photos and your traffic is phone photos. Query provenance means where the query image came from. Measure the gap, because it is large and invisible if you never split on it:

recall@100, same model, same index

catalog image re-uploaded as the query      0.912      30% of traffic
user phone photo, good light                0.784      70% × 0.30
user phone photo, low light or motion       0.631      70% × 0.40
user photo with 3+ objects, no crop         0.417      70% × 0.30

user side  = 0.30(0.784) + 0.40(0.631) + 0.30(0.417)  =  0.6127
overall    = 0.7 × 0.6127 + 0.3 × 0.912                =  0.703

Reporting 0.912 because that is what the gold set contains overstates the shipped system by 21 points. The user side is 40% low-light because that is what photographing furniture in a cafe looks like, and the weighted 0.703 is unreproducible without those sub-weights. The fix is the query-photo pairs in both training and evaluation, and a per-provenance slice on every experiment.

Online

Offline numbers rank models; one online number decides whether the product is better. The metric is attributed purchase within 7 days of a visual search, per search (attributed meaning credited back to the search that led to it). Not CTR, and for a specific reason: near-duplicate flooding raises CTR@1, because 12 photos of the same appealing chair make a very clickable first result, while destroying the session because the user has nothing to compare. A metric that improves when the product gets worse is not a metric.

Sizing an experiment on the fast proxy at its 31% baseline against a 2% relative effect needs about 89,000 searches per arm. At 600k searches/day split 50/50, that is roughly seven hours. The primary metric at a 2.1% purchase base rate needs about 20x that. So the fast proxy gates the ramp and the primary metric gates the launch.

Serving architecture

Every component now takes its place on one request path. The path runs top to bottom; the four highlighted stages carry the numbers derived above (encoder, ANN, reranker, and the dedupe most systems get wrong):

flowchart TD
    U(["User photo"]) --> DEC["Decode + EXIF orient<br/>downscale to 640"]
    DEC --> MOD{"Moderation<br/>+ CSAM hash"}
    MOD -->|block| REJ(["Reject"])
    MOD -->|pass| DET["Object detector<br/>largest salient box"]
    DET --> CROP["Crop + pad to 224"]
    CROP --> ENC["ViT-B/16 encoder<br/>35 GFLOP · 8 ms<br/>version-pinned"]
    ENC --> QC{"Query cache<br/>key: pHash + model_version"}
    QC -->|"hit 9%"| RES
    QC -->|miss| ANN["ANN · 5 shards<br/>efSearch 128 · 3 ms<br/>k' = 800"]
    ANN --> FIL["Eligibility filter<br/>region · stock · policy"]
    FIL --> DD["Content-cluster dedupe<br/>image hash, not product_id<br/>612 listings -> ~211 clusters"]
    DD --> RR["Reranker · GBDT<br/>60 cross-features<br/>200 candidates · 9 ms"]
    RR --> DIV["MMR diversity<br/>lambda = 0.75"]
    DIV --> RES(["Top 20"])
    RES --> LOG[("Impression log<br/>+ propensity + model_version")]

    style ENC fill:#1d3557,color:#fff
    style ANN fill:#2d6a4f,color:#fff
    style RR fill:#bc6c25,color:#fff
    style DD fill:#9d0208,color:#fff

Walking the path: decode the JPEG and apply EXIF orientation; run two safety checks (a moderation classifier and a CSAM hash lookup, CSAM being child sexual abuse material, checked by exact fingerprint against a known list, not a model); crop the largest salient box to 224x224 (padding, not stretching); encode to 512 numbers in 8 ms; check a query cache keyed on the pHash plus model version that answers 9% of requests; search five ANN shards in parallel for k' = 800 ids; filter out anything out of region, stock, or policy (612 survive); dedupe those to ~211 distinct products; rerank with the GBDT; run MMR for a slate whose items are not all the same thing; log every impression with its propensity (the probability that item was shown in that slot, which makes the log usable for training).

Three load-bearing details:

• The cache key includes the model version. Without it, an encoder rollout silently serves v1 results to v2 queries. A cache is the one place a v1 vector outlives a v2 rollout.
• Dedupe happens before the reranker, and keys on the image-hash cluster, not product_id. The 612 survivors collapse to 211 clusters, so reranking first would waste 66% of the expensive stage on candidates about to be thrown away. And k' is 800, not the mean requirement of 580, so the 200 rerank slots are full after both filter attrition and the collapse.
• The impression log carries the propensity because the cold-start exploration slot is randomized; without the probability each item was shown with, that slot produces logs you cannot debias later.

Index lifecycle

The request path assumes a current index. It gets that way through three independent flows:

flowchart LR
    NEW["New/updated item"] --> IQ[["Ingest queue"]]
    IQ --> EMB["Encode<br/>batch 256"]
    EMB --> POOL["Pool photos -> one<br/>vector per product"]
    POOL --> INS["HNSW insert<br/>1.2 ms/vector"]
    INS --> LIVE[("Live index<br/>alias -> v_n")]
    DELETED["Delisted item"] --> TOMB["Tombstone<br/>filtered post-search"]
    TOMB --> LIVE
    REBUILD["Encoder upgrade"] --> FULL["Full re-encode<br/>200M images"]
    FULL --> V2[("Shadow index v_n+1")]
    V2 -.->|"atomic alias swap<br/>ALL shards or none"| LIVE

    style LIVE fill:#2d6a4f,color:#fff
    style V2 fill:#bc6c25,color:#fff

A new item is encoded in batches of 256, pooled to one vector per product, inserted at 1.2 ms/vector, and answerable within minutes. A delisting is tombstoned, not removed; the compaction schedule eventually reclaims it. An encoder upgrade re-encodes all 200M products into a shadow index and promotes it by a single atomic alias swap across all shards or none. That last flow is the dangerous one; doing it gradually is the drift incident below.

Two nodes mean more than they look. The encode-batch-256 node is the ingest queue’s reason to exist: the encoder that costs 8 ms at batch 1 reaches full GPU utilization only at batch 256, so ingest buffers and serving does not, two deployments of one model with opposite latency contracts. And the tombstone node is not a delete: anything that counts index size, including the per-shard memory alarm, has to count tombstones separately from live vectors, or it will read healthy at 61% dead.

One query, end to end

Every figure below was derived above:

2.1 MB phone photo of a chair, shot in a cafe

  upload + EXIF orient + decode + resize          45 ms   -> 640 px long edge
  moderation 0.003, CSAM hash miss                 6 ms   -> pass
  detector: 3 boxes, largest is the chair @ 0.71  18 ms   -> crop + pad to 224
  ViT-B/16, exact-product head, L2-normalized      8 ms   -> 512-d fp16
  query cache: miss                                       -> search
  ANN, 5 shards, ef 128, k' = 800                  3 ms   -> 800 listing ids
  eligibility: region, in-stock, policy            6 ms   -> 612 survive
  content-cluster dedupe, 612 / 2.9 mean                  -> 211 clusters
  rerank, GBDT, 60 cross-features, top 200         9 ms   -> scored
  MMR, lambda = 0.75                               2 ms   -> 20 clusters
                                                  -----
                                                   97 ms

  top-1 cosine 0.86 >= tau_exact 0.83  ->  EXACT-PRODUCT LANE
  slot 1 expands the winning cluster back into its 4 seller listings,
  ranked by price and seller quality
  slots 2-20: the next 19 clusters, one listing each

Both decisions the user can feel were derived, not chosen: 0.86 clears 0.83 because a wrong “same item” was priced at $14 against $0.60, and the slate holds 20 distinct products because dedupe keys on the content cluster, not product_id. Collapsing to clusters and then expanding the winner is not a contradiction: dedupe guarantees 20 things to compare, and the exact lane guarantees the one you asked for is shown with every way to buy it.

Scale and cost

Full reindex. Re-encoding 200M images at 34.9 GFLOP each is about 6.5 GPU-hours; the HNSW build is 25 minutes across five shards. Priced out, a full reindex is about $32 of compute plus 30 TB of reads, well under a day. So the constraint on upgrading the encoder is not compute. It is the atomicity of the swap and the 2x memory during the dual-index window (234 GB × 2 ≈ 469 GB while both versions are resident); provision for it or you cannot roll back.

Steady state. At 600k searches/day and a 25 QPS peak, the compute is trivial (the encoder is under one GPU, the ANN and reranker are negligible), so the bill is almost entirely RAM for the index:

15 machines × $0.62/hr × 720 h/month          =  $6,696/month
+ reserved 0.5 H100 for encode                 =  $  900
+ ingest re-encode (3% weekly churn)           =  $ 2.11
+ compaction, one shard every 5 days           =  $ 4.69
                                                 ------------
                                                 $7,603/month

88% of the cost is RAM holding an index that answers 25 queries per second, and at the end of each compaction cycle a tenth of it holds items nobody can buy. The tombstones point at the compaction schedule (the difference between 15 machines and the 39 an uncompacted year ends at). The RAM points at binary quantization: dropping the index to 42 GB is three machines instead of fifteen, and with 205 GB of NVMe for the rescore payloads at $16/month the bill falls to about $2,258/month, a 3.4x win for two points of index recall and 2 ms of random reads. Quote it that way, not as free. The reserved half-GPU buys availability, not throughput, the same shape as the RAM line one order down.

Failure modes

The system breaks in five ways, each with the same anatomy: a mechanism, a measurement that detects it, and a control that prevents it. None is fixed by a better model.

Near-duplicate flooding

The most common visible failure (the slate fills with the same chair twenty times) is a data-modelling error, not a model error. A photo-level index (920M vectors, one per catalog image) returns:

QUERY   a photo of a mid-century walnut lounge chair

  ranks 1-9    SKU 88213, nine catalog photos of the same chair
  ranks 10-14  SKU 88213 listed by four other sellers, plus one crop
  ranks 15-20  SKU 90447, six photos

distinct physical products in the top 20:   2
distinct product_ids in the top 20:         7
CTR@1:                             0.41   <- UP. The metric approves.

The failure inside the failure is the second-to-last line. A product id is per listing, so four sellers listing the same chair are four ids and a re-cropped upload is a fifth. A dedupe keyed on product_id collapses the nine catalog photos into one row and leaves the other six, and the guardrail metric then reads 7 against a target of 20 and raises nothing. The dedupe key and the guardrail metric must be the same key, and it must be a key on the content (the perceptual-hash / 0.97-cosine cluster already built for mining).

The index contains photos; the user is shopping for products. Three fixes, and they are not alternatives:

• Index at the product level. Pool a SKU’s photos into one vector. A mean of L2-normalized embeddings works; view-clustered centroids (cluster a SKU’s photos, keep one centroid per viewpoint, score against the best-matching one) work better for products shot from very different angles, at 2-3 index entries per SKU. This removes the intra-SKU flood and shrinks the index 4.6x, from 920M photo vectors to the 200M product vectors used throughout.
• Over-retrieve and dedupe across sellers. Pooling fixes duplicates within a SKU, but the same chair listed by five sellers is still five entries. Measured, a distinct product occupies a mean of 2.9 listings and a p95 of 5.4, so ask the index for k' = 800 and collapse down. (That is the mean requirement plus filter headroom, not a p95 guarantee, which would cost 200 × 5.4 = 1,080.)
• Diversify the final slate, because two genuinely different SKUs can still look nearly the same. MMR picks the slate one item at a time, scoring each candidate by relevance minus its similarity to whatever is already picked, with lambda weighting the two sides:

score(i) = lambda · rel(i)  -  (1 - lambda) · max_{j in selected} sim(i, j)

MMR does not rescue a broken dedupe key. Solving for the lambda at which MMR is indifferent between one more clone and a genuinely different SKU gives lambda* = 0.318 on this slate: every clone outranks the different product until lambda drops below 0.318, which is 68% of the weight on diversity, which shreds relevance on slates that were never redundant. The crossover always lands that low because relevance gaps between near-identical candidates are small while redundancy gaps are order 1. Tune lambda (0.75 ships) against both the diversity and proxy metrics, but it is not a substitute for deduping on content.

The dedupe-then-MMR sequence, with the key that matters made explicit:

def dedupe_and_diversify(candidates, k=20, lam=0.75):
    """Collapse content-level duplicates, then MMR for visual diversity.

    candidates: ordered dicts with content_key, rel, and L2-normalized vec.
    The dedupe key is content_key (the perceptual-hash / 0.97-cosine cluster),
    NOT product_id: one chair sold by five sellers carries five product ids and
    survives a product-id dedupe intact.

    Two mechanisms in sequence: dedupe removes the *same* product appearing many
    times (an indexing artifact); MMR removes *different* products that look the
    same (a genuine ranking choice). MMR does not substitute for the dedupe.
    """
    seen, pool = set(), []
    for c in candidates:
        if c["content_key"] not in seen:
            seen.add(c["content_key"])
            pool.append(c)

    selected = []
    while pool and len(selected) < k:
        best, best_score = None, None
        for c in pool:
            redundancy = max(
                (sum(a * b for a, b in zip(c["vec"], s["vec"])) for s in selected),
                default=0.0,
            )
            score = lam * c["rel"] - (1 - lam) * redundancy
            if best_score is None or score > best_score:
                best, best_score = c, score
        selected.append(best)
        pool.remove(best)
    return selected

The red dress that returns red curtains

This failure teaches the most transferable lesson in the chapter: a model uses whatever shortcut the training task leaves available, so the way to fix a bad representation is usually to change the task, not the architecture.

QUERY   a red bodycon dress, photographed indoors

rank  item                          cos    category
  1   red midi dress                0.81   dresses
  2   red wrap dress                0.79   dresses
  3   crimson blouse                0.77   tops
  4   red velvet curtain panel      0.76   home/window
  5   red satin pillowcase          0.74   home/bedding

The mechanism is not “the model is confused.” It is that the training task never required shape: the augmentations had crops, flips, and blur but no color jitter; the negatives were random items, which differ from the anchor in color 97% of the time; so a representation that encodes only the dominant color histogram already solves the contrastive task, and gradient descent finds the cheapest sufficient solution.

The diagnostic is a linear probe on the frozen embedding. All three targets are nominal (a color bucket, a leaf category, a SKU id), so the metric is top-1 accuracy against chance plus NMI (normalized mutual information, how much the prediction tells you about the truth, scaled to [0,1] and comparable across targets of different cardinality), not R^2. R^2 is a share of variance and needs an interval scale; on a nominal target it is just a function of the integer codes somebody assigned, and permuting those codes changes it while changing nothing about the model.

linear probe on the frozen embedding

target                 classes   chance   top-1 accuracy    NMI
dominant colour bucket      12    0.083            0.87    0.71
leaf category            4,800    0.0002           0.52    0.44
SKU identity             1,000    0.001            0.22    0.19

State this carefully: “the embedding is 87% a color descriptor” is a share-of-variance claim the numbers do not support. The defensible version is that a linear probe reads the color bucket off the frozen embedding 87% of the time against an 8.3% chance rate and reads the SKU 22% of the time, and NMI orders them the same way. NMI is on the table because 0.52 over 4,800 categories is a far larger lift over chance than 0.87 over 12, so accuracy alone is not comparable across cardinalities.

Both fixes are about the task, not the architecture:

• Color-jitter the augmentations, which makes color an invariance and forces the model onto shape and texture. Watch the exact-product metric for products that differ only in colorway, and keep color as an explicit reranker feature, where it is a decision you control instead of a shortcut.
• Mine attribute-stratified negatives: items that share the anchor’s dominant color but differ in category, so a color-only representation gets them wrong and stops being a local optimum.

After both, on the same eval and probe sets: color accuracy 0.87 -> 0.31, category 0.52 -> 0.74, SKU 0.22 -> 0.61, and recall@100 on the low-light user-photo slice 0.631 -> 0.752. The general lesson: augmentations define what the model must ignore, and negatives define what it must distinguish. Any property in neither is one the model is free to use as a shortcut.

Embedding drift on reindex

The worst failure, because it is both total and silent: search returns nothing useful, and every dashboard stays green.

INCIDENT
  encoder v2 deployed to the query path at 14:02
  index still holds v1 vectors for 62% of shards (rolling backfill)

  cos(v1(x), v2(x)) for the SAME image, 1,000 items:  0.31

  recall@100 on the affected shards:   0.71 -> 0.04
  zero-result rate:                     0.3% -> 22%
  alert that fired:                     none for 41 minutes
                                        (latency normal, error rate 0, QPS normal)

A rolling backfill replaces the index machine by machine while serving, which is how almost every other data migration is safely done and is exactly wrong here. Two encoder versions are two different vector spaces, not two nearby versions of one. A retrain from a different seed lands on unrelated axes; dimension 7 of v2 has nothing to do with dimension 7 of v1, and no rotation reconciles them. Three controls:

1. The model version is part of the index identity and every cache key, so a v2 query cannot structurally reach a v1 shard.
2. Build the new index beside the old one and swap an alias atomically across all shards, or not at all. A rolling backfill of an embedding index is a bug, not a deployment strategy.
3. Alert on recall against a frozen probe set, evaluated continuously (200 queries with known answers, run every minute). It is the only monitor that fires here; latency, error rate, and QPS all look perfect.

Cold start for new items

Cold start is serving something with no history, and visual search only half has it. Retrieval is fine: a new SKU is fully retrievable the moment its photo is encoded, unlike a collaborative-filtering system (which recommends by “people who liked this also liked that” and knows nothing about an item nobody has touched). The problem is the reranker, whose most predictive features are engagement counters that are null for a new item:

reranker feature importance (gain):
   ctr_30d              0.24     null for new items
   purchase_rate_30d    0.19     null
   return_rate          0.11     null
   embedding cosine     0.14
   attribute overlap    0.09
   -> 54% of the model's gain is unavailable on a new listing

A GBDT sends nulls down a default branch learned from training data, and that default is “behaves like a low-CTR item,” so new items rank below where they belong, get no impressions, accumulate no engagement, and the cold start becomes permanent. Both fixes attack the null, because the ranker is behaving correctly on the data it was given:

• Shrink toward a category prior instead of imputing null. Shrinkage blends an item’s thin evidence with its group average, weighted so the group dominates when the item has little data and fades as it accumulates: ctr_hat = (clicks + alpha·ctr_cat) / (impressions + alpha). With alpha = 200 and ctr_cat = 0.031, a new item with 50 impressions and 4 clicks lands at 0.041, within 0.001 of an established item at 0.042, while their raw rates (0.080 vs 0.042) are 2x apart. Fifty impressions is not evidence, and shrinkage says so in the units the ranker consumes. alpha is the number of impressions at which the item’s own data outweighs the prior; pick it by cross-validation.
• Reserve an exploration slot. One position in the top 20 goes to a random eligible new item, with the propensity logged. Because the choice was random, this is the only uncontaminated data about new items, and it costs about 0.4% of conversion.

Query-side failures

These live in the photograph itself, and hold the largest single recall win in the chapter:

Summary of the failure modes

Alternatives considered and rejected

Four names first: CLIP is a pretrained model trained on images paired with captions, so it knows what a picture is about but not which particular chair it is; zero-shot means using it untrained. SIFT and ORB are classical pre-neural methods that match distinctive keypoints between two images geometrically. A cross-encoder reads the query and candidate together in one pass, far more accurate than two independent vectors, which is exactly why it cannot be indexed. nprobe is IVF’s search-time dial for how many clusters to scan.

Conclusion

The load-bearing decisions in a visual search system are not the network:

• Route on a threshold derived from cost, not tuned from a sweep. A 23:1 asymmetry between a wrong “same item” ($14) and a missed match ($0.60) forces a 96% confidence bar, which calibrates to tau_exact = 0.83 and splits traffic into an exact lane (32%) and a style lane (68%). The style lane is a consequence of the threshold, not a separate feature.
• The loss is the system. Learn a relation, not a class. Random negatives stop teaching after one epoch; a mined hard negative carries roughly 7,300x the gradient, while a 16x bigger batch buys only ~14x. Mine, but from a semi-hard band and after deduping the catalog, because the hardest negative is a mislabeled positive 16% of the time.
• Labels are free; annotation buys only the 3,000-query gold set. The evaluation trap is provenance: a catalog-photo gold set overstates the shipped system by 21 points.
• The index is memory, and memory is the bill. 88% of the steady-state cost is RAM. HNSW tombstones force a compaction schedule that doubles as the takedown SLA; binary quantization with rescore is a 3.4x cost cut for two points of recall, not free.
• The visible failures are data-modelling and versioning, not model quality. Dedupe on content not product_id; fix the color shortcut by changing the task; never roll a new encoder into an old index.

One line to remember: in visual search, the network is the easy part, and every decision that matters is about which wrong answers the model sees and which duplicates the index shows.

Further reading

• Schroff, Kalenichenko, Philbin, FaceNet: A Unified Embedding for Face Recognition and Clustering (2015): triplet loss and instance-level metric learning.
• Wu, Manmatha, Smola, Krähenbühl, Sampling Matters in Deep Embedding Learning (2017): why negative sampling dominates, and semi-hard mining.
• van den Oord, Li, Vinyals, Representation Learning with Contrastive Predictive Coding (2018): the InfoNCE loss.
• Yi et al., Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations (2019): the logQ correction for in-batch negatives.
• Malkov, Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (2018): HNSW.
• Dosovitskiy et al., An Image Is Worth 16x16 Words (2020): the Vision Transformer.
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (2021): CLIP.

Related on this site: feature engineering for lookahead leakage, metrics for cost-matrix thresholds, and vector index internals for ANN structures.

Next: the image-privacy chapter, an offline pipeline where the operating point falls out of a legal cost asymmetry, and mAP is the wrong number to report.