How to design a RAG system

Retrieval-augmented generation (RAG) answers a question by first retrieving relevant passages from your own documents, then including those passages in the model’s prompt. The answer is built from text the model reads at query time, not from what it learned during training.

In this lesson, we’ll take the system view of RAG: not the retrieval algorithm, but the running service around it. We’ll build the pipeline that turns a changing set of files into a searchable index, price what that index costs in memory and dollars, and hunt the failures that produce fluent, confidently wrong answers while raising no error. By the end you’ll be able to size each tier from a corpus, name the number that forces the next move, and build a detector for every silent failure before it reaches a customer.

That last point is the spine of the lesson. Almost every failure here is silent: no exception, no error log, no latency spike. A system with no detector for them looks healthy until a customer finds their contract text inside someone else’s answer. So each mechanism is paired with the detector that catches it when it breaks.

The RAG-for-agents lesson covers a different question: retrieval as a tool the model chooses to call, how to merge rankings, how to cut documents into passages, and when to reach for GraphRAG. That lesson decides whether to search; this one owns the index, the pipeline, and the on-call rotation. Where an idea appears in both, this lesson gives the price and the failure mode and links back for the mechanism.

(GraphRAG is an index built over an extracted entity graph instead of over passages. It exists so that questions whose answer lives in no single passage, “what themes recur across all incident reports?”, can be answered by aggregating over the graph instead of fetching a passage that does not exist.)

Two paths on two clocks

The system has two paths that run on different clocks, and nearly every design decision belongs to one or the other.

• Index time runs once per document version. It is allowed to take seconds.
• Query time runs once per question. It must finish inside a person’s patience.

Index time turns one document into searchable rows. A 40 KB PDF of refund-policy version 7 becomes eleven rows, one per chunk:

refund-policy#0   "Refunds are available within 30 days of..."   [0.031, -0.114, ...]
refund-policy#1   "To start a refund, contact support and..."    [0.088,  0.002, ...]
...
refund-policy#10  "Enterprise agreements may override this..."   [-0.007, 0.240, ...]

Each row carries the chunk id, the chunk text, a 1,024-number vector, and stamps the rest of the lesson depends on: corpus_version, tenant_id, ingested_at, source_modified_at, and the id of the embedding model that produced the vector.

Query time turns a question into a cited answer:

IN    "how long do I have to request a refund on an annual plan?"

      retrieved and ranked (scores are cosine similarities, defined below):
        refund-policy#3   0.71
        billing-faq#2     0.64
        refund-policy#8   0.58

OUT   "Annual plans may be refunded within 30 days of the renewal date,
       minus any usage [refund-policy#3]."

The bracketed chunk id is a citation, and it makes several failure detectors possible later. On this path, retrieval is only about 1.5% of the response time and 11% of the cost; generation is the rest. That is the opposite of where most teams spend their attention.

Vocabulary, defined once

About twenty terms carry the lesson. They are defined here and referred back to as needed.

The data

• Corpus: the collection of documents you search: the wiki, the PDF archive, the ticket history.
• Token: the unit models count text in, roughly four characters of English. A 500-token chunk is about 375 words.
• Chunk: one passage cut out of a document, a few hundred tokens long. Documents are too long to retrieve or read whole, so at index time each is cut into pieces. Retrieval returns chunks, never whole documents.
• Tenant: one customer of a system that serves many. The defining requirement: one tenant’s chunks must never reach another tenant’s answer.

Dense retrieval: searching by meaning

An embedding is a fixed-length list of floating-point numbers, 1,024 of them for a typical model. The model was trained on pairs of related texts, so texts a human would call related come out pointing in similar directions. The dimensions have no names; nothing guarantees dimension 47 means anything describable. It is called dense because nearly all 1,024 slots hold a nonzero number.

Dense retrieval is three steps: embed every chunk once at index time and store the vector; embed the incoming question the same way at query time; return the chunks whose vectors point most nearly the same direction as the question’s.

“Most nearly the same direction” is cosine similarity: the cosine of the angle between two vectors. It is 1.0 when they point the same way, 0 when unrelated, and it ignores vector length. The store that holds those vectors and answers direction queries is a vector index.

Sparse retrieval: searching by words

The other family scores a chunk on which of the query’s exact terms appear in it, and on how rare those terms are across the corpus. It is called sparse because the natural representation is one slot per vocabulary word, and almost every slot is zero. BM25 is the standard formula (“Best Matching 25”, the twenty-fifth ranking function in a line of 1990s IR research), and the default in Lucene, Elasticsearch, and OpenSearch.

The two families fail in opposite directions, which is why both exist:

• Dense answers “how do I get my money back” from a page that only says refund. Sparse cannot; no query word appears.
• BM25 finds ERR_4021 or tier-2, exact strings a dense embedding averages into noise. Dense cannot; averaging a passage into 1,024 numbers destroys rare literals.

Running both and merging is hybrid search. The standard merge is reciprocal rank fusion (RRF): score each result 1/(k + its rank) in each list, then add the two scores. RRF uses only positions, never the raw scores, so it never has to make a cosine similarity and a BM25 score comparable, which they are not.

Approximate search, and two things called “recall”

Scoring the query against all N stored vectors is exact, and its cost grows linearly with N. An ANN index (approximate nearest neighbour) inspects only a small fraction of the corpus and returns almost the true nearest vectors, about 340x faster at the scale priced later.

Two different metrics measure two different things, and confusing them is common:

• Index recall: the fraction of the true nearest neighbours the approximate search actually returned. 0.96 means 4 in 100 were missed, with no error raised.
• Recall@k: the fraction of test questions whose gold passage (the chunk a human labelled as containing the answer) appears in the top k.

An index can sit at 0.96 index recall while Recall@5 is 0.40, because returning the true nearest neighbours does not help when those neighbours are the wrong passages.

Two more labelled metrics appear later:

• nDCG@k: normalized discounted cumulative gain. Sum each relevant result’s usefulness, discounted by how far down it sits, divided by the score of the perfect ordering. 1.0 is perfect; position is penalized.
• MRR: mean reciprocal rank, the average of 1 / (rank of the first relevant result). It cares only how fast the user hits something useful.

Reranking: a second, slower pass

The retrieval stage runs over millions of chunks, so it uses a bi-encoder: query and chunk are embedded separately and compared by cosine. That separation is what lets chunk vectors be computed once and stored forever.

Reranking is a second pass over the shortlist, take the top 50, score them properly, keep the best 5. The scorer is a cross-encoder: it reads query and chunk concatenated, in one forward pass, so every query word can attend to every chunk word. It is more accurate because it sees the pair, not two independent summaries of it, and it cannot be precomputed because the pair does not exist until the query arrives. That is why it is slow, and why it runs on 50 candidates, not 10 million.

The generation side

• Grounding: every claim in the answer traces to a retrieved passage, not to the model’s memory.
• Hallucination: a claim that does not so trace. Fluent, plausible, unsupported.
• Citation resolution: the check that each [refund-policy#3] the model emitted is a chunk id that (a) exists and (b) was actually in the context you gave it. An unresolvable citation is the cheapest hallucination detector there is.

Two version stamps, which are not the same stamp

Two version counters run through the system. They answer different questions, and swapping them breaks a cache in a way nothing reports.

corpus_version is what makes an alias flip atomic and what fusion asserts on: two rows with different corpus_version came from two different builds and must never be ranked together. A full rebuild moves both counters; one editor saving one page moves only index_version.

The retrieval cache must key on index_version, the per-write one. Key it on corpus_version instead and the cache keeps serving the pre-edit ranking until the next full rebuild. You can check which your system does in five minutes: edit one document and watch whether the counter moves.

Operational words

• SLO: service level objective, the target you promise (“a new document is findable within ten seconds”). SLI, the measured number you compare against it.
• p50 / p99: percentiles of a latency distribution. p50 is the median; p99 is the value only 1% of requests exceed.
• Idempotent: doing it twice leaves the same result as doing it once. This is what makes retrying safe.
• Upsert: write a row, replacing it if it already exists.
• CDC: change data capture. Read a database’s own write log, so you learn about every insert, update, and delete in order, instead of periodically re-scanning and guessing what moved.

Two numbers decide the architecture

The shape of the whole system follows from two facts about the corpus:

1. How fast must a new document become findable? This is the freshness SLO, the promised gap between a write in the source system and that write being retrievable.
2. How big is the corpus, and how fast does it change?

The table below sorts the options into four rungs. You climb one only when the corpus or the SLO forces it, and each rung adds a system to operate.

Two rows need unpacking. pgvector is a PostgreSQL extension that stores embeddings in an ordinary table column and searches them with SQL; because the vector lives in the same transaction as the row it describes, it kills an entire bug class (covered under alternatives). Quantized vectors means storing each number in fewer bits than a full 32-bit float; it turns out to be the second-largest cost lever in the system.

The first row is a real answer, not a placeholder. Below about 100,000 tokens the entire corpus fits in one prompt. The cheapest correct retrieval system is no retrieval system.

Corpus size tells you how many machines. The SLO tells you how many systems. A one-hour SLO lets you rebuild the index nightly and diff, one batch job, no streaming ingest, no online insert, no version-stamped read path. A ten-second SLO forces all three into existence as separate systems with separate failure modes. A third question, how many tenants?, applies only sometimes: past tens of thousands, isolation stops being a policy choice and becomes a memory-budget calculation.

The table assumes documents whose text is what you search, arriving at a rate one pipeline can absorb, in a corpus where a single passage usually contains the answer. Each assumption breaks a row:

• Aggregate questions (“recurring themes across all incident reports”) have no single passage containing the answer. Top-k retrieval cannot build it from passages that do not exist; you need a graph index, not a bigger passage index.
• Answers that must be exact and current (order status, account balance) want a database query against the system of record, not an approximate search over a copy that goes stale on every write.
• A small corpus with enormous query volume inverts the cost model: generation, not indexing, becomes the entire bill.

The indexing pipeline is a streaming system

Every tutorial writes the index-time path as a loop:

for doc in corpus:
    for chunk in split(doc):
        index.add(embed(chunk))

That is a batch job, correct exactly once, at t=0. The moment the corpus changes, three things become true that a loop over a list cannot express:

1. You must know what changed. A delete is not the absence of an insert. A full re-scan cannot tell “this document was removed” from “the crawler missed it.”
2. You must be idempotent. Queues are at-least-once, workers crash mid-document, and a retry must not double-insert.
3. You must be ordered per document. Two edits to the same document in flight means the older one can land last.

So ingest is a stream processor. The unit of work is a document version; the operations are upsert and delete; the key is doc_id, with chunk ids derived deterministically from it.

Two terms the pipeline diagram assumes:

• An index alias is a name queries resolve through. Searches go to vectors-live, which currently points at concrete index v9. Repointing it is an alias flip, and the flip is atomic: every query before it reads v8, every query after reads v9, no query sees a half-built index.
• A shadow index is a complete second index built quietly off to the side while the live one keeps serving. You build it, audit it, then flip the alias to it. That flip is the only moment the live system changes, which is what makes a bad rebuild something you discard instead of repair under pressure.

flowchart TD
    SRC[("Source of truth<br/>CMS · S3 · Confluence · DB")] -->|CDC / webhook| BUS["Change bus<br/>doc_id · version · op"]
    BACK["Backfill / reindex job"] --> Q2

    BUS --> Q1["Incremental queue<br/>priority lane · 40 workers"]
    BACK -.->|never shares a lane| Q1

    Q1 --> P
    Q2["Backfill queue<br/>rate-limited · 200 workers"] -->|"targeted re-ingest<br/>(a few thousand docs)"| P["Fetch · parse · chunk"]
    P --> E["Embed<br/>cache: sha256 text + model_id"]
    E --> U["Atomic replace by doc_id<br/>one write, not two"]
    U --> LIVE[("Live index<br/>corpus_version N")]
    Q2 -->|"full reindex<br/>(whole corpus)"| SHADOW[("Shadow index<br/>corpus_version N+1")]
    SHADOW -->|recall audit passes| FLIP{{"Atomic alias flip<br/>vectors AND bm25 together"}}
    FLIP --> LIVE

Following one document through the pipeline makes the stages concrete. An editor saves refund-policy in Confluence. A connector reading the write log (or a webhook the source fires at you) emits one small message onto a durable, ordered log carrying only doc_id, version, and op, no content, which is why this stage costs half a second instead of seconds. An incremental worker picks it up, fetches and parses (2 s for a PDF, 0.05 s for HTML), and hits the embed cache keyed on sha256(chunk text) + model_id, 8 of the 11 chunks are unchanged, so only 3 are re-embedded. Then one atomic replace by doc_id into the live index, stamped corpus_version N.

Two things the diagram asserts that a box cannot say:

• The dotted arrow means “never”. No backfill message is ever enqueued into the incremental lane, by construction, not convention. That absence is the fix for the queue explosion derived below, drawn as negative space.
• The pool sizes look backwards and are not. Backfill has five times the workers (200 vs 40), yet it is the backfill lane that is rate-limited, and the limit is applied at the embedding tier, not the worker tier. Workers are cheap and spend most of their time waiting on network and PDF parsing; embedding capacity is the resource both lanes actually contend for. Throttle the contended resource, not the idle one.

The lexical (BM25) index is invisible in the drawing because it shares the parse stage entirely: built from the same parsed text, against the same corpus_version, and it lives behind the same flip. When a full reindex is done, one atomic alias flip moves the vector index and the BM25 index together, never one and then the other.

The pipeline rests on three assumptions, each a real deployment somewhere:

1. The source can tell you what changed (CDC or webhook). If it cannot, detection becomes a full re-scan interval, two orders of magnitude larger than every other term combined, and deletes stop being detectable at all.
2. Documents are individually re-processable. If a “document” is a 400-page manual whose chunks only make sense together, per-document atomic replace is still right but the unit of work has to become a section.
3. The embedding function is pure and stable for a given model id. That is what makes the cache free money and retries safe, and it stops being true the moment a hosted provider silently updates a model behind an unchanged name (see the version-skew failure below).

What each stage costs, and how each one fails

The last two columns carry the weight here. The failures shown in bold have no exception attached; everything else throws, gets retried, and never reaches a customer.

P/2 is the expected wait when you poll every P seconds instead of subscribe: if the write lands at a random moment in a cycle, the average wait is half a cycle. The idempotency key is what makes a retry a no-op, not a duplicate: retry the fetch and the content hash says “same bytes”; retry the embed and (chunk_hash, model_id) hits the cache. The three bolded failures raise nothing: a missed delete leaves a document retrievable forever, a scanned PDF that parses to an empty string indexes zero chunks and reports success, and version skew ranks perfectly happily.

The rule that removes an entire bug class

Chunk ids are a deterministic function of (doc_id, chunk_index), and a document update replaces every chunk under that doc_id atomically, the write fully happens or does not happen at all, with no state a query could observe in between.

The rule is emphatically not “upsert the chunks I produced this time.” That difference produces an orphan chunk: a row still in the index whose text no longer exists in any live document. Watch what happens when a document gets shorter:

t0  doc "refund-policy" v1  ->  11 chunks: refund-policy#0 .. #10
t1  edited and shortened    ->   9 chunks: refund-policy#0 .. #8
    upsert overwrites #0..#8.  #9 and #10 are never touched.

t2  query "how long do I have to request a refund?"
    top-3:  refund-policy#9   0.74   (v1 text: "...within 60 days...")
            refund-policy#3   0.69   (v2 text: "...within 30 days...")
            billing-faq#2     0.61

    -> the model gets two contradictory passages under the SAME source id,
       from the SAME document, and picks one. No exception, no log line,
       the citation looks perfect.

The obvious fix, delete_by_doc then insert, is the right semantics and the wrong number of calls. It is two writes, and a worker that dies between them leaves the document absent, not stale, gone.

Absent is strictly worse: a stale passage at least answers with old text; a missing document answers nothing and is indistinguishable from a crawler that never saw the file. So the replace has to be one atomic operation, and that is a requirement on the storage layer, not something the caller can arrange:

If you can only pick the ordering, pick insert-then-delete. Stale-and-duplicated is a ranking problem you can detect and repair; absent is a silent hole that looks exactly like “the crawler never saw this document.”

Freshness: the staleness window

Freshness becomes tractable once you turn “how fresh is the index?” into a number. The staleness window W is the elapsed time from a write in the source system to the moment a query can retrieve the new content. It is a sum, because each stage finishes before the next begins:

W = t_detect + t_queue + t_parse + t_chunk + t_embed + t_upsert + t_refresh

Six terms are the pipeline stages above, in order. The seventh, t_refresh, is the one people forget: the delay between a write being durable in the index and being visible to a search. It is real in every system that batches writes into segments before making them searchable, and it is a hard floor you cannot optimize below.

Run identical code two ways, differing only in how change is detected:

PUSH (webhook or CDC)                 PULL (poll every 15 min)
  t_detect    0.5 s                     t_detect   450 s   (P/2 expected)
  t_queue     0.3 s                     t_queue      0.3 s
  t_parse     2.0 s   (PDF)             t_parse      2.0 s
  t_chunk     0.01 s                    t_chunk      0.01 s
  t_embed     0.6 s                     t_embed      0.6 s
  t_upsert    0.05 s                    t_upsert     0.05 s
  t_refresh   1.0 s                     t_refresh    1.0 s
              ------                                ------
    W_p50  =  4.5 s                       W_p50 = 454 s  (7.6 min)

The polling interval dominates every other term by two orders of magnitude, 450 s of detection against ~4 s of everything else. Shaving 200 ms off embedding is meaningless while you poll every 15 minutes. Freshness is one architectural decision, push or pull, not a tuning exercise. (The push column assumes the source emits a durable event for every write including deletes; a webhook dropped during a deploy is a document silently never updated. The pull column assumes writes arrive uniformly in a cycle, which makes the expected wait P/2; a corpus edited in a morning burst is much worse than the average.)

The queue term is the one that explodes

Every term above is a steady-state number. One is not bounded in steady state, and it is the one that breaks the SLO in production. The M/M/1 model (random arrivals, random service, one server) gives the expected wait as W_q = rho / (mu · (1 - rho)), where rho = lambda/mu is utilization, the fraction of capacity in use. At rho = 0.20 the wait is ~12 ms; even at rho = 0.95 it is under a second. The (1 - rho) denominator is what makes it explode as rho approaches 1: work arrives exactly as fast as it clears and the backlog never drains.

A reindex drives rho past 1 on purpose. Trigger a bulk reindex against a single shared pool of 40 workers serving both lanes, the configuration everyone starts with:

bulk reindex enqueues              2,000,000 documents
40 workers x 0.5 doc/s (PDF-heavy)  =  20 docs/s
drain time = 2e6 / 20  =  100,000 s  =  27.8 hours

For those 27.8 hours, every incremental update’s staleness window is also 27.8 hours, because updates sit in the same queue behind two million backfill messages. Against the 4.5-second push median that is missing the target by a factor of ~22,000. And users report “search is stale,” not “the backfill is running,” so the wrong team diagnoses it slowly.

The fix is structural, not a bigger pool (doubling workers only halves 27.8 h to 13.9 h, still four orders past target):

• Two queues, separate pools. Incremental never waits behind backfill, by construction. This is why the backfill pool can be 200 workers while incremental keeps 40: once the lanes cannot mix, making backfill faster no longer makes freshness worse.
• A rate limit on the backfill lane so it cannot consume more than a fixed share of embedding capacity, the resource the lanes still share after the pools are separated.
• Backfills write to a shadow index and swap, so a full rebuild never contends with the live index at all, and a bad backfill is discarded, not repaired.

Vector index internals

Three structures compete, and the choice reduces to two questions:

flowchart TD
    A["Choosing a vector index"] --> B{"Fewer than ~1M vectors?"}
    B -->|Yes| FLAT["Flat: exact scan, recall 1.0, no tuning"]
    B -->|No| C{"Continuous writes and deletes?"}
    C -->|Yes| HNSW["HNSW: cheap online insert,<br/>plus scheduled compaction for deletes"]
    C -->|"No, rebuilt periodically"| IVF["IVF: cheap rebuild via k-means"]

Throughout, N is the number of vectors and d the dimension (numbers per vector). The running example is N = 10M, d = 1024, so every figure is comparable.

Flat — the baseline

A flat index stores the vectors in a list and, at query time, computes cosine similarity against every one. Nothing to build, nothing to tune. Its cost is memory traffic, not compute: the machine reads every stored byte to score it. At N = 10M, d = 1024, fp32 (4 bytes each), that is 10M × 1024 × 4 ≈ 41 GB scanned per query; at ~100 GB/s effective bandwidth, ~410 ms per query. Scale down and it stops being alarming, at 100k vectors the same arithmetic gives ~4.1 ms.

Flat is not a toy. It is the correct answer below roughly 1M vectors, and the only index whose recall is exactly 1.0. Reaching for HNSW at 200k vectors buys a build step, a tuning surface, and a delete problem in exchange for latency you did not need. The assumption underneath is that ~100 GB/s figure: every flat latency scales inversely with it, and if the vectors do not fit in RAM at all, the disk-bound scan is an order of magnitude slower and flat stops being viable far below 1M.

IVF — partition and probe

IVF (inverted file index) buys speed by looking at only part of the corpus. At build time, k-means clustering finds nlist centre points and assigns each vector to its nearest one. At query time it compares the query against the nlist centres only (a few thousand comparisons), then exhaustively scans the nprobe cells whose centres came out nearest, ignoring the rest. Inserts are cheap: assign the new vector to its nearest centroid and append.

With nlist ≈ 4·√N ≈ 12,650 and nprobe = 32, a query scans N · nprobe / nlist ≈ 0.25% of the corpus, so it pays 0.25% of flat’s 410 ms, about 1.0 ms, roughly 400x less memory traffic.

nprobe is a pure speed/recall dial with zero memory cost: it is a query-time count of how many cells to open, not a stored structure, so you can turn it up for a compliance sweep and back down for interactive traffic without rebuilding. IVF’s weakness is that the centroids were fitted to the corpus as it looked when k-means last ran; as new documents arrive in regions the old centroids do not cover, recall at a given nprobe drifts down. The fix is re-running k-means, cheap, but still a rebuild.

HNSW — memory per vector

HNSW (hierarchical navigable small-world graph) is what most production systems use. Every vector is a node linked to a few of its nearest neighbours; search is a walk, start somewhere, hop to whichever neighbour is closer to the query, repeat until no neighbour improves. “Small world” means mostly-local links plus a few long ones, which gives short paths between any two points (the way six degrees of separation works). “Hierarchical” stacks nodes into layers: a sparse top layer with long links down to layer 0, which holds every point; search enters at the top, descends toward the query, and at layer 0 does a beam search of width efSearch (keeping the best efSearch candidates alive at once).

Three parameters: M, the neighbour links per node on upper layers (default 16); layer 0 gets 2M links because it holds every point and hosts the final beam search; and efSearch, the beam width, a query-time dial with no memory cost, exactly like nprobe.

The memory budget rests on one number, bytes per vector, worth being able to reproduce. Node levels are drawn from an exponentially decaying distribution, P(level ≥ l) = M^-l, so the expected number of upper layers is the geometric series Σ M^-l = 1/(M-1) = 1/15 ≈ 0.067. Upper layers are therefore nearly free, and almost the entire cost is the vector payload plus layer 0:

vector payload      d x 4 bytes         1024 x 4        =  4,096 B
layer-0 neighbours  2M ids x 4 bytes    32 x 4          =    128 B
upper-layer links   0.067 x 16 x 4                      ≈      4 B
per-node bookkeeping (level, offset, external id)       ≈     16 B
                                        M=16, d=1024      4,244 B  ≈ 4.24 KB

All KB/GB here are decimal (1,000 / 1e9 bytes), the units RAM is sold in. Multiplying out: 10M chunks ≈ 42 GB (one machine); 100M chunks ≈ 424 GB (not one machine). That single computation is the entire reason quantization exists.

Quantization: the same vectors in fewer bits

Quantization stores each of the d numbers in fewer bits, accepting a small error in the similarity scores. Four options:

• fp16: a 16-bit float, half the bytes with a shorter mantissa.
• int8 scalar: 256 integer steps per dimension; you need each dimension’s min/max across the corpus to map back to real values.
• Matryoshka: a training scheme (named after the nesting dolls), not compression: the model is trained so the first 256 of its 1,024 dimensions are already a usable embedding, which is why truncating a Matryoshka vector is legitimate and truncating an ordinary one is destructive.
• Binary: one bit per dimension, just the sign; comparisons become bit operations (fast) but most magnitude information is gone (poor raw recall).

Each row below re-runs the same budget with a different payload line. The recall column reads as deltas against this table’s own fp32 baseline, not as absolute numbers, because every row is measured at one fixed graph configuration to isolate what changes when the representation changes.

So fp16 costs nothing, int8 costs about one point, and binary-plus-rescore costs about two. Both factors matter and neither is a constant: halve the dimension or the corpus and the whole GB column halves, so carry the bytes/vector across problems, never the GB.

Binary plus rescore is the interesting row: 15x less RAM with most of the recall back. The binary pass is only a candidate generator; rescoring takes its top 200, fetches the full-precision vectors for just those, and re-sorts, about 2 ms of random reads from NVMe (fast flash storage). The point is that the full vectors still live somewhere: you moved them out of RAM onto storage far cheaper per gigabyte, trading 424 GB → 28 GB of RAM for 2 ms of latency and two points of recall.

The comparison, on the columns that decide it

Put the structures side by side and the surprise is that the two columns everyone reads, latency and recall, are nearly identical across every non-flat row, so they decide nothing. The two that decide are online insert and delete.

A tombstone is a node marked deleted and filtered from results but still physically present and still used as a routing hop by searches passing through, the standard way graph indexes handle deletion, because truly removing a node would break the links of every neighbour that pointed at it.

Choose HNSW when writes are continuous; choose IVF when the corpus is rebuilt periodically. HNSW inserts online at 1.2 ms/vector but cannot truly delete without a rebuild; IVF needs a fresh k-means when the distribution drifts but rebuilds cheaply. Published benchmarks plot recall against QPS (queries per second) and hide both of these columns, which is why people pick from the wrong two.

One vendor question hides here: “HNSW cannot truly delete” is really a property of most implementations. Whether a deleted node’s slot is ever reclaimed, and whether segment merges compact the graph for you, differs between hnswlib, FAISS, Lucene, and every managed service. If the service compacts on merge, the tombstone budget below does not apply to you; if it does not, that budget is mandatory. Ask explicitly. This is the single thing managed vector databases document least well.

Why recall below 1.0 is usually fine

Losing 4 points of index recall does not cost 4 points of answer quality, because the ANN stage is not the last stage and the stage after it is also lossy. What matters end to end is whether the gold passage survives into the handful of passages the model sees, through two stages, so the probabilities multiply:

P(gold in final 5) = P(gold in ANN top-50) x P(reranker keeps it | it was in the 50)

exact flat:    1.000 x 0.87  =  0.870
HNSW ef=64:    0.960 x 0.87  =  0.835      -3.5 points end to end
HNSW ef=128:   0.990 x 0.87  =  0.861      -0.9 points end to end

The 0.87 is the reranker’s measured Recall@5 at 50 candidates (the ceiling: even a perfect front end loses 13% of gold passages here). These are upper bounds on the loss, for two reasons: 0.960/0.990 are Recall@10 figures standing in for Recall@50, which is strictly higher; and the 0.87 was itself measured with an approximate front end, so using it here charges part of the ANN loss twice. A 4-point ANN recall loss becomes at most a 3.5-point end-to-end loss, in exchange for a 340x latency reduction (flat’s 410 ms against HNSW’s 1.2 ms). Buying the 0.9-point version instead costs 2.5 ms and 3% more RAM per vector. Many questions are answerable from several passages, so losing the single annotated gold chunk frequently costs nothing measurable.

This all assumes ranking is the deliverable, that a slightly wrong ordering degrades an answer instead of invalidating it. Three jobs where that is false:

1. Exact-neighbour semantics, deduplication, near-duplicate detection, entity resolution (deciding two records refer to the same real person or company), plagiarism. The nearest neighbour is the answer, so a miss is a wrong answer. Use flat, or ANN with exact rescoring and a generous candidate pool.
2. Recall is the deliverable, legal discovery, compliance sweeps, safety audits. “We searched and found nothing” has to mean it, and 0.96 recall means one document in 25 was never looked at.
3. Selective filters, the big one. Selectivity is the fraction of the corpus a filter admits. Post-filtering retrieves the top k by vector, then drops what fails the filter. For a tenant owning 0.01% of the corpus, retrieving k = 50 then filtering leaves 50 × 0.0001 = 0.005 expected survivors, so 99.5% of that tenant’s queries return zero results and the system reports “no relevant documents,” which is a lie indistinguishable from an empty corpus. Pre-filtering (traverse only eligible nodes) fixes the count but breaks HNSW another way: the graph’s links encode nearness in the full corpus, so a greedy traversal keeps hopping to ineligible neighbours, stalls, and degenerates toward a linear scan, graph overhead for scan performance. Below roughly 1% filter selectivity you must partition instead of filter: physically separate the eligible vectors into their own index, so the predicate chooses which structure to search, not what to discard afterwards.

Sharding and multi-tenancy

These look like separate topics and are one: both decide which vectors are physically stored together, and the selectivity result above already proved that physical grouping, not query-time filtering, is the only thing that works below 1% selectivity.

Sharding

Sharding splits one logical index across machines, each holding a disjoint shard. The choice is the splitting rule, and the bill is tail latency whenever a rule requires asking more than one shard.

Fan-out means sending the query to every shard and merging, because any shard might hold the best match. The merge is cheap; the waiting is not, because a fan-out query finishes only when its slowest shard finishes. If one shard exceeds 5 ms with probability 0.01, then across 16 shards the chance that at least one does is 1 - 0.99^16 = 0.149. Your p99 becomes your shards’ p85, a one-in-a-hundred event on one machine becomes one-in-seven once sixteen must all behave. Mitigate with fewer, larger shards (fewer chances to draw a slow one) or hedged requests (when a shard is late, fire a duplicate at a replica and take the first response). The derivation assumes independent shard latencies; correlated slowness (a GC pause, a noisy rack) makes the real number better, because the slow events coincide on one query.

Multi-tenancy

Where do each customer’s vectors physically live? Three arrangements, two wrong:

The hybrid is the answer, and the reason is the filter arithmetic, not economics. Tenants above ~0.5% of the corpus get a dedicated index. Tenants below it live in a shared index physically partitioned by tenant, so the tenant predicate (tenant_id = "acme") selects a partition instead of filtering results after the fact. Get it backwards and small tenants get empty answers while large tenants are fine, a bug that reproduces only for the customers least able to report it precisely.

Two details worth stating. First, the 50,000-tenant rejection rests entirely on that ~50–200 MB per-index overhead, which is the cost of an index process before it holds a single vector (graph metadata, segment maps, BM25 dictionary, runtime heap floor, per-replica reservations). It is implementation-specific and spans more than an order of magnitude: at 200 MB, 50k tenants is 10 TB of pure overhead (impossible); at 2 MB it is 100 GB (merely expensive). So measure it on your stack, create one empty index, measure resident memory, create a hundred, divide. The shape of the argument survives either way; the tenant count where it bites is yours to measure.

Second, the two thresholds (1% selectivity for filter breakdown, 0.5% for the dedicated cutoff) differ deliberately. Setting the cutoff below the breakdown point means every tenant left in the shared index is unambiguously in the regime where filtering fails, which removes the judgement call. Anyone in the shared index is partitioned; anyone big enough that a filter might have survived already has their own index.

The leak

This is the failure that ends careers. One argument goes missing, six months apart:

# the original, correct call
retriever.search(query_vec, top_k=10, filter={"tenant_id": ctx.tenant})

# six months later, a "global knowledge base" feature ships
retriever.search(query_vec, top_k=10)          # filter omitted; defaults to None

RESULT: chunks from 340 tenants, ranked by cosine similarity.
        No exception. No error log. Latency is NORMAL - slightly better.
        The agent cites them with source ids that look internal and correct.
        Discovered three weeks later by the customer whose contract text
        appeared inside another customer's answer.

The control is to make the unscoped call unrepresentable, not merely validated: tenant scope is a constructor argument on the retriever, and search has no tenant parameter at all. The same shape closes a second hole, a query embedded by one model searched against an index built by another (version skew, below), but only if the check lives where the vector is. A constructor that compares two model-id strings and then hands you a search(query_vec) accepting any vector is decoration, because the constructor never sees a vector.

class Embedding(NamedTuple):
    """A vector that carries the id of the model that produced it.
    An untagged vector cannot be checked against anything, so it is not accepted."""
    model_id: str
    vec: tuple

class ScopedRetriever:
    def __init__(self, index, tenant_id, encoder):
        if not tenant_id:
            raise ValueError("refusing to construct an unscoped retriever")
        self._index, self._tenant, self._encoder = index, tenant_id, encoder

    def search(self, query, top_k=50):
        # Control 1: no parameter widens the tenant scope.
        # Control 2: the model-id check, and it must be HERE, not in __init__,
        #            because __init__ never sees a vector.
        q = self._encoder(query) if isinstance(query, str) else query
        if not isinstance(q, Embedding):
            raise TypeError("a bare vector carries no model id; nothing can "
                            "check which space it lives in")
        if q.model_id != self._index.embed_model_id:
            raise ValueError("query vector from a different model than the index")
        return self._index.search(q.vec, top_k=top_k, partition=self._tenant)

Delete the two lines comparing q.model_id and a v1 vector ranks happily against a v2 index with no error, the entire version-skew incident in four lines, and why the check cannot live in __init__.

Two things this control assumes, both walk-aroundable. A wildcard tenant id like "*" satisfies the non-empty check and constructs cleanly; if your storage treats that as “all partitions,” the guard was passed, not defeated. And the underlying index still has a wide API one attribute access away, so isolation holds only as long as nothing else in the process is handed the raw index. The structural control raises the cost of the mistake enormously; a canary tenant covers the cases where somebody paid it, a fake customer holding poison documents with sentinel strings (nonsense tokens like zq7-canary-4418 that cannot occur naturally), plus a nightly job asserting those strings never appear in any other tenant’s results. This failure has no other detector: no error, no latency anomaly, no unusual log line.

Reranking in the serving path

The query-time path end to end, with every stage priced. It settles the most common false economy in retrieval systems: that reranking is too slow to afford. This is the classic single-pass path: embed once, retrieve once, rerank once, generate once. (In the agentic variant the model may call retrieval zero, one, or several times per question; multiply the retrieval subtotal by the expected number of calls, the RAG-for-agents lesson derives a band of roughly 0.6x to 6x.)

flowchart LR
    Q(["Query"]) --> QC{"Query-embedding cache<br/>saves 18 ms on a hit"}
    QC -->|hit 15-40%| RC
    QC -->|miss| EMB["Embed · 18 ms"]
    EMB --> RC{"Retrieval cache · saves 73 ms on a hit<br/>key: query · tenant · filters · index_version"}
    RC -->|hit 20-35%| CTX
    RC -->|miss| PAR["ANN 1.2 ms + BM25 8 ms<br/>in parallel · max = 8 ms"]
    PAR --> RRF["RRF fusion · 0.1 ms"]
    RRF --> RR["Cross-encoder rerank<br/>50 candidates · 45 ms"]
    RR --> CTX["Assemble 3-5 passages · 2 ms"]
    CTX --> LLM["Generate · 4,650 ms<br/>TTFT 650 ms · decode 4,000 ms"]
    LLM --> A(["Answer + citations"])

The two caches and the reranker are the stages worth spending on; generation is the stage that already has all the money. Written out for the worst case (every cache missed, the right case to design an SLO against):

query embed (hosted API)                18 ms
ANN search (HNSW, ef=64)               1.2 ms
BM25 (Lucene, 10M docs)                  8 ms   parallel with ANN, so max(1.2, 8) = 8
RRF fusion                             0.1 ms
rerank 50 candidates x 250 tokens       45 ms   <- the "expensive" stage
assemble context                         2 ms
                                       -------
retrieval subtotal                    73.1 ms

LLM time to first token (3.5k prompt)  650 ms
LLM decode, 400 tokens               4,000 ms
                                       -------
end to end                           4,723 ms   retrieval = 1.5 %

The 45 ms rerank figure is derived, not quoted. A transformer forward pass costs about 2 floating-point operations per parameter per token, so 50 candidates × 250 tokens = 12,500 tokens through a 300M-parameter cross-encoder is 2 × 3e8 × 12,500 = 7.5 TFLOP; at ~300 TFLOP/s effective that is ~25 ms, plus batching and one network hop, ≈ 45 ms. Reranking is 45 ms against a 4.7-second response, 1% of latency and the single largest quality lever in the pipeline. The objection “reranking is too slow” measures the wrong denominator. The real questions are whether the reranker batches across concurrent queries and whether it fits before the first streamed token.

The parameter that actually matters is rerank depth, how many candidates the reranker is given, not whether it exists. What counts is where the curve bends, which the table below makes visible:

50→100 doubles cost for 2 points; 50→20 halves cost but gives back 6. Fifty is not magic. It is where the curve bends for this corpus, and you find it by plotting it. Three legitimate reasons to skip reranking entirely: nDCG@5 measures identical with and without it, latency-critical autocomplete, or a UI that shows three results with a large top-1 margin.

Two assumptions move the whole argument. First, generation dominates (650 ms to first token, 4,000 ms decode), which stops holding for a system that answers with one extracted sentence, and the latency argument then has to be re-run. Second, every recall number above was measured on your corpus, because the bend at 50 depends on how many near-misses your ANN stage puts between rank 20 and 50. Import the method, never the numbers.

Caching: three caches, three risk profiles

Four things can be cached; three of them should be. Which is which turns on a single property of the cache key, not on staleness tolerance.

The two embedding caches are free money because embed(text, model) is a pure function (output depends only on inputs, no hidden state, no clock), so identical text has identical vectors forever, as long as the model id is in the key. On a re-ingest where 5% of documents changed, the cache turns a full re-embed into a 5% re-embed, a 20x reduction on the dominant ingest compute.

The other two caches can go wrong, and one field decides which. index_version is in the retrieval key and not in the answer key:

t0   query "what is the refund window?"
     retrieval -> [refund-policy#3 @ index_version 4471]  ->  answer "30 days."   both cached
t1   legal updates the policy to 14 days. Ingest runs. index_version 4471 -> 4472.
t2   same query:
     retrieval cache: key holds 4471, current is 4472  ->  MISS, re-retrieves.   CORRECT
     answer cache:    key is query_hash alone           ->  HIT, returns "30 days."   WRONG
                      ...with a citation to a document that now says 14,
                      which makes the wrong answer look verified.

A retrieval cache entry is invalidated by the very thing that would make it wrong. An answer cache entry is not, because the answer’s correctness depends on documents the key does not contain. Adding index_version to the answer key fixes it and collapses the hit rate to near zero on any live corpus, the honest signal that the cache was never viable. The one answer-cache key that works is the retrieved chunk ids plus their content hashes: it invalidates exactly when the evidence changes, and survives index rebuilds that changed no content (where an index_version-keyed cache would throw everything away for nothing).

Two assumptions decide whether those hit rates are yours: queries must repeat verbatim after normalization (weak for long questions, and it can vanish in an agentic system where the model rewrites the question before searching), and index_version must advance on every change (if it only moves on full rebuilds, the retrieval cache silently inherits the answer cache’s problem, with no signal).

The cost model

Two bills (indexing a corpus once, and answering a question) and reading them shows the money is not where anyone expects. Prices are US dollars at the rates listed; the method survives the rates changing, so carry the derivations, not the totals.

Ingest, per 1M documents

Assume an 8-page document ≈ 5,300 tokens, structural chunking at ~500 tokens → 11 chunks/document, so 1M documents → 11M chunks → 5.5B tokens.

(OCR, optical character recognition, is the expensive branch of parsing: reading characters out of an image, which is what a scanned page is. MTok is a million tokens, the unit hosted APIs bill in.)

Parsing is 98% of ingest cost. Embedding is 1%. That inverts what almost everyone expects and redirects the optimization: the win is “do not OCR the 60% of the corpus that is already digital text,” which drops hosted parsing from $12,000 to ~$4,800, not “find a cheaper embedding model.” And self-hosting embeddings ($13 vs $110) does not matter: five GPU-hours of work does not justify operating a service. Self-host embeddings only when you re-embed continuously (a 100M-chunk corpus with a monthly model refresh), not when you ingest once.

Almost all of this rests on the corpus being PDFs, the 98% is a statement about document format, not retrieval. It holds for scanned archives, contracts, and manuals; it collapses for wiki pages or source code, where parsing is nearly free and embedding dominates. State the format assumption before the number, because the number is worthless without it.

Storage, recurring

Ingest is a one-time bill; storage arrives every month, and it is where quantization stops being a tuning detail.

11M chunks x 4.24 KB (HNSW fp32, d=1024)     =  46.6 GB RAM
effective RAM price ≈ $7.37 / GB-month       (a 256 GB instance at $2.02/hr, ~200 GB usable)
46.6 GB x $7.37                              =  $343 / month
x2 replicas for availability                 =  $687 / month

with int8 (1.17 KB/vector):  12.9 GB x $7.37 x 2  =  $190 / month     3.6x cheaper

(Replicas are copies of the whole index on separate machines, so one failure does not take search down; two is the usual minimum and doubles the bill.) Put next to the one-time bills, storage passes the whole embedding bill in the first week and the entire $12,150 ingest bill in about 18 months. Quantization is not a micro-optimization. It is the second-largest cost lever, behind only the generation context size. The assumption making RAM the recurring bill is that the index must be memory-resident to hit its latency target; a disk-backed index changes this line by an order of magnitude and the latency budget by more, exactly the trade binary-plus-rescore makes deliberately.

Query time, per 1,000 queries

Assume 5 passages × 500 tokens, a 1,000-token system prompt, a 50-token question, 400 output tokens, on claude-sonnet-5 at $3/$15 per MTok (input/output; output costs more everywhere because it is produced one token at a time). Input is 1,000 + 2,500 + 50 = 3,550 tokens, so per query 3,550/1e6 × $3 + 400/1e6 × $15 = $0.0167.

Generation is 89% of query-time cost. The entire retrieval stack is 11%, and vector search itself is 0.1%. Four consequences, in priority order:

1. Tuning efSearch to save 0.5 ms optimizes 0.1% of the bill. Do not spend the week.
2. Cut from 5 passages to 3. That removes 1,000 input tokens for ~$3.00/1k, 16% of the total bill from one config change, and frequently gives better answers, because the marginal passages were diluting attention.
3. Self-host the reranker above roughly 5M queries/month ($2.00/1k → $0.017/1k, worth 10.7%). The threshold is judgement, not the raw crossover: the gap is the cost of running and monitoring a service you did not previously operate.
4. Prompt caching, which, as specified, saves nothing. See below.

Prompt caching is a hosted-API feature: mark a stable prefix (the breakpoint) and repeat requests beginning with exactly those tokens are billed at a fraction of the input rate, because the provider reuses the work on that prefix. Two rules govern whether it fires: the prefix must be byte-identical across requests, and it must clear a minimum length or it silently does not cache. The floors are not even monotonic across a vendor’s generations (512 on claude-opus-5, 1,024 on claude-sonnet-5, 4,096 on claude-haiku-4-5). The model priced here is claude-sonnet-5, and the stable prefix assumed above is 1,000 tokens, 24 short of the 1,024 floor, so nothing caches, cache_creation_input_tokens comes back 0, no error is raised, and the real saving is $0.00. Pad the prefix past the floor (24 tokens of boilerplate) and the ~14.8% is real. One more constraint: retrieved passages vary per query, so put them after the breakpoint. Before it, they invalidate the cache on every request.

The optimization order at query time is context size, then rerank vendor, then never the ANN parameters, and prompt caching only enters the list once the prefix clears the floor. This is the cheapest instance in the lesson of a control that looks like it works and does not: it raises no error, returns a well-formed response, and delivers none of the assumed saving. Its detector has the same shape as every silent failure here. Assert the thing you assumed, on every call: alert if cache_creation_input_tokens is 0 on a request whose prefix you believe is cacheable.

Two assumptions move the whole table: one retrieval and one generation per question, at $3/$15 per MTok. An agentic loop that searches three times pays the generation line roughly four times over, making the retrieval stack a smaller fraction still. A cheaper generation model compresses the 89% toward the retrieval stack, at which point hosted reranking becomes the largest line. Re-run the table whenever either changes; do not carry the percentages.

Evaluation: three layers, and nobody builds the third

The generation metrics: faithfulness is the fraction of the answer’s claims actually supported by the given passages (the direct inverse of hallucination); answer relevance is whether the response addresses the question asked, not a nearby one; citation accuracy is citation resolution, scored: each [chunk-id] must exist, must have been in context, and must support the sentence it is attached to.

Diagnose retrieval first, because it is a dependency. If Recall@10 is 0.4, the right passage is absent from context 60% of the time, and no prompt change makes a model cite what it cannot see. Tuning generation first is optimizing a downstream stage against a broken input.

The system layer is the one that gets skipped and the one that matters, because every silent failure below is invisible to the first two layers. Four measurements:

1. Nightly ANN recall audit. Sample 1,000 production queries, run each against the live HNSW index and against a flat index over the same vectors, and score |top10_ann ∩ top10_exact| / 10. Average over the sample for last night’s index fidelity, one number to trend. This is the only way you learn the index degraded after a rebuild with different parameters, after months of deletions, or after a distribution shift. Gate the alias flip on it: a shadow index that misses the bar does not go live, and that it missed is the alert.
2. Staleness p99. Stamp ingested_at on every chunk and carry source_modified_at from the origin; the p99 of the difference is your freshness SLI. A backfill starving the incremental lane shows up here and nowhere else.
3. Model-id assertion on the search path (every query, refusing a foreign vector, not once at construction, where nothing has a vector yet).
4. Orphan and tombstone counts. Orphans are chunks whose doc_id no longer exists upstream or whose chunk_index exceeds the current version’s count; the tombstone ratio is the fraction of graph nodes that are deleted-but-resident.

The retrieval and generation layers need a labelled set that looks like production, questions with known-correct passages, and if the team wrote them, they are the questions the team would ask, which is systematically not the traffic. The system-layer metrics need no labels at all, which is exactly why they keep working the day the labelled set goes stale.

Failure modes

Every failure below produces a fluent, well-formatted, confidently wrong answer with a citation that looks correct. None raises an exception, shows a latency anomaly, or is caught by the retrieval or generation metrics. Each comes with the mechanism, the detector, and the control.

Half-swapped alias after a reindex

What happens when the atomic flip is applied to two indexes one at a time. Between 04:12 and 05:40 the two names point at different corpus versions:

02:00  backfill builds vector index v9         (completes 04:12)
04:12  alias  vectors-live  ->  v9             [flipped]
       alias  bm25-live     ->  v8             [BM25 rebuild still running]

04:30  query "annual plan refund window"
       dense  -> refund-policy#3   (ids from corpus v9)
       BM25   -> refund-policy#9   (ids from corpus v8; deleted in v9)
       RRF fuses both. The v8 id is looked up in the v9 content store -> KeyError,
       swallowed by a try/except in the assembler, passage silently dropped.
       Recall for that query: half. No alert. The answer cites the surviving passage.

Every id in that ranking came out of your own index moments ago; an id that does not resolve means the two halves of the system disagree about what exists. Two indexes over one corpus must flip atomically, or ids from different corpus versions will be fused into one ranking. Controls: a single corpus_version both indexes are built against, a flip that moves both or neither, and a hard assert at fusion time that every candidate id carries the same corpus version. Never a swallowed KeyError in the assembler, a missing document is a page, not a shrug.

Embedding-model version skew

The widest blast radius and the weakest symptoms: an entire corpus disappears from search while every dashboard metric stays green. You upgrade from embed-v1 to embed-v2, same dimension so nothing raises; new documents enter with v2 vectors, old documents keep v1 vectors, same index.

Two independently trained models share no coordinate system: dimension 47 of v1 and dimension 47 of v2 encode unrelated features. A dot product between a v2 query and a v1 document is a sum of 1,024 products of unrelated coordinates, each as likely positive as negative, so they cancel: the result is a random number with mean ≈ 0 and spread ≈ 1/√d = 1/√1024 = 0.031. Nothing about it is an error; it is a perfectly well-formed number that means nothing. So cross-version similarities pile into a tight band around zero while same-version similarities spread across 0.3–0.9:

query encoded with v2; index holds 8M v1 vectors + 400k v2 vectors
cos(q_v2, d_v1):  mean 0.002, sd 0.031, max observed 0.14
cos(q_v2, d_v2):  mean 0.31,  sd 0.12,  max observed 0.89

top-10 for EVERY query: 10 of 10 documents ingested after the migration date.

The pre-migration corpus behaves exactly as if deleted, and nothing raises an error. Recall on the 95% that is old falls from 0.94 to about 0.05.

Detection is a distribution check, never an exception: the p50 top-1 cosine (drops when queries stop finding good matches) and, sharper, the age distribution of retrieved documents, a vanished corpus shows up as every result being newer than one date, the migration date. That costs one histogram and is unmistakable.

Controls, and only the first is real: (1) the model id is part of the index identity and part of every vector, the index is named chunks_embed-v2, vectors travel as (model_id, vec), and the retriever compares them on every search; checking it once at construction cannot stop the vector that matters, because the constructor never sees one. (2) Blue/green: build the v2 index completely, run the recall audit against it, only then flip, never migrate in place. (3) Dual-write during the build so the v1 index stays live and correct up to the flip. The re-embed itself is cheap ($110 hosted, $13 self-hosted); it is the second index copy and the operational care people are trying to avoid, and that is exactly the wrong thing to economize on.

Chunk-boundary loss

Created at index time, paid at query time, the reason chunking is a design decision, not a preprocessing detail. A fact split across two chunks, and a retriever that behaves perfectly and still loses it:

document "Billing FAQ", section 4
  chunk 7 (ends):    "... Annual plans may be refunded within"
  chunk 8 (begins):  "30 days of the renewal date, minus any usage."

query: "how long do I have to get a refund on an annual plan?"
dense top-3:
  1. billing-faq#7   0.71   has "annual", "refunded", "plans"
  2. billing-faq#2   0.64   general refund overview
  3. billing-faq#8   0.58   has "30 days", "renewal", but NOT "refund"

context budget admits 2 passages  ->  chunk 8 is dropped
answer: "Annual plans are refundable, though the specific window is not stated."

The retriever ranked the chunk with the query’s vocabulary first, exactly as designed. The failure was a splitter that cut a sentence in half, putting the lexical signal (“annual”, “refunded”) and the content (“30 days”) on opposite sides of a boundary. Four fixes, cheapest first:

Contextual chunking is the largest reported single-technique retrieval win, at $2,200 / $12,000 = 18% of the parsing bill, an easy yes, not a research project. Its detector is Recall@10 healthy but faithfulness low.

Deleted documents that stay retrievable

The one with legal consequences, and it accumulates silently over months. HNSW has no true delete: a deleted node is tombstoned (still resident and still traversed as a routing hop, merely filtered from results) because removing it would orphan the links of everything that pointed at it. At 5% deletion per month with no compaction, 1 - 0.95^12 = 46% of graph nodes are tombstones after a year:

memory       1.85x     (= 1 / 0.54) RAM for deleted vectors
latency      +~40%     traversal walks through dead nodes
recall       drifts    the link structure was fitted to the original point set
compliance   FAILED    the vector still exists in RAM and in every snapshot

The last row is the one that matters. A right to erasure (the legal obligation, under regimes such as the GDPR, to actually delete a person’s data on request) is not satisfied by a tombstone, the vector is still in RAM, still in every backup, still reconstructible. The control is scheduled compaction: rebuild the shard from only its live vectors, tied to an SLA (the contractual version of an SLO) matching your erasure commitment. Track requests to specific shards so you rebuild the two that matter, not all forty.

Summary

For every one of these, the detector is the design decision, because none announce themselves:

Alternatives considered and rejected

Several choices are rejected only at this scale and are right at another, so the last column names the crossover where there is one.

Conclusion

The design collapses to a few load-bearing facts:

• Two clocks. Index time may take seconds; query time may not. Corpus size decides how many machines; the freshness SLO decides how many systems.
• The pipeline is a stream, not a loop. CDC into an ordered queue, idempotent stateless workers, unit of work is a document version, and the update is one atomic replace-by-doc, never a bare upsert (orphan chunks) or delete-then-insert (an absent document).
• Memory is the recurring bill. HNSW costs ~4.24 KB/vector, so 100M chunks is 424 GB; quantization is the second-largest cost lever, behind only the generation context size.
• Generation dominates cost and latency. Retrieval is ~11% of the bill and ~1.5% of the latency; the optimization order is context size, then rerank vendor, then never the ANN parameters. Reranking is 45 ms for the single largest quality lever.
• Isolation is structural. Below ~1% selectivity, partition instead of filter; make the unscoped, wrong-model call unrepresentable, not merely validated.
• Every serious failure is silent. For each one, the detector is a design decision, the model id on every vector, the nightly ANN-vs-flat audit, staleness p99, orphan and tombstone counts, the canary tenant. A system with no detector for these looks healthy until a customer finds their contract in someone else’s answer.

One line to remember: the index is the easy part; the silent failures are the system, so build the detector beside every mechanism, never after it.

flowchart TD
    SRC[("Source of truth")] -->|CDC / webhook| ING["Streaming ingest<br/>idempotent · ordered · atomic replace-by-doc"]
    ING --> IDX[("Vector + BM25 index<br/>corpus_version · model_id · tenant_id<br/>HNSW quantized")]
    ING -.->|full rebuild| SHADOW[("Shadow index")] -.->|recall audit gates the flip| IDX

    Q(["Query"]) --> HYB["Hybrid retrieve<br/>ANN + BM25 -> RRF"] --> RR["Cross-encoder rerank<br/>top 50 -> top 5"] --> GEN["Generate + cite<br/>~89% of cost"] --> ANS(["Answer + citations"])
    IDX --> HYB

    IDX --> DET["Detectors<br/>ANN-vs-flat audit · staleness p99<br/>orphan / tombstone counts · canary tenant"]

Further reading

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020), the original RAG paper.
• Malkov & Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (IEEE TPAMI, 2018), HNSW.
• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009).
• Cormack, Clarke & Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (SIGIR 2009), RRF.
• Jégou, Douze & Schmid, Product Quantization for Nearest Neighbor Search (IEEE TPAMI, 2011); Johnson, Douze & Jégou, Billion-Scale Similarity Search with GPUs (FAISS, 2017).
• Kusupati et al., Matryoshka Representation Learning (NeurIPS 2022).
• Anthropic, Introducing Contextual Retrieval (2024), contextual chunking.

Next: Face Generation.