Retrieval-augmented generation (RAG) answers a question by retrieving passages from your own documents and putting them in the prompt; the answer is built from text read at query time. The index is the easy part, almost every real failure is silent (no exception, no latency spike), so pair every mechanism with its detector.
Two clocks, two sizing rules
- Index time: runs once per document version, may take seconds.
- Query time: runs once per question, must fit human patience. Retrieval is ~1.5% of latency and ~11% of cost; generation is the rest.
- Corpus size decides how many machines; the freshness SLO (gap from write to retrievable) decides how many systems.
| Corpus | Change | Freshness SLO | Architecture |
|---|---|---|---|
| <100k tokens | rare | any | No retrieval; cache whole corpus in prompt |
| <5M chunks | daily | minutes | pgvector beside source table |
| 5M–100M | continuous | seconds | Dedicated ANN, streaming ingest, quantized |
| >100M | continuous | seconds | Sharded, tiered, per-tenant partitions |
Retrieval building blocks
- Chunk: one passage (~few hundred tokens; 1 token ≈ 4 chars). Retrieval returns chunks, never whole docs.
- Embedding: fixed-length vector (~1,024 floats); related texts point the same way. Dense = search by meaning via cosine similarity (1.0 same, 0 unrelated).
- BM25 = sparse, exact-term/rarity scoring; finds literals like
ERR_4021that dense averages into noise. - Hybrid search = run both, merge with RRF (
1/(k+rank), positions only, so incomparable scores never mix). - Reranking: second pass, take top 50 → keep best 5, using a cross-encoder (reads query+chunk concatenated; slow, cannot precompute). Retrieval uses a bi-encoder (embedded separately, precomputable).
- Index recall (no labels) = fraction of true neighbours the ANN returned. Recall@k (needs gold labels) = fraction of test questions whose gold passage is in top k. Also nDCG@k, MRR. An index at 0.96 index recall can have Recall@5 = 0.40.
Pipeline = stream, not a loop
- Unit of work is a document version; ops are upsert/delete keyed on
doc_id. Needs: know what changed (a delete ≠ absent insert), be idempotent, be ordered per doc. - Detect via CDC or webhook (push), not polling (
P/2expected wait dominates all else). - Atomic replace-by-doc, one write. Bare upsert on a shortened doc leaves orphan chunks (old text, ranked and cited); delete-then-insert risks an absent document (worse: looks un-indexed). If forced to pick, insert-then-delete.
- Full rebuilds go to a shadow index, audited, then one atomic alias flip moving vector + BM25 together on one
corpus_version. - Two stamps:
corpus_version= a build (set once per run);index_version= current contents (moves on every write). Retrieval cache MUST key onindex_version.
Vector index internals
| Index | RAM/vec (d=1024) | p50 @10M | Recall@10 | Insert | Delete |
|---|---|---|---|---|---|
| Flat | 4.10 KB | 410 ms | 1.000 | append | true delete |
| IVF nprobe=32 | 4.11 KB | 1.0 ms | 0.95 | cheap | tombstone |
| HNSW M=16 ef=64 | 4.24 KB | 1.2 ms | 0.96 | 1.2 ms/vec | tombstone only |
- Flat = exact scan, correct below ~1M vectors. HNSW (small-world graph,
Mlinks,efSearchbeam) for continuous writes; IVF (k-means cells,nprobe) for periodic rebuilds. Latency/recall are near-identical across non-flat rows; insert and delete decide. - HNSW ≈ 4.24 KB/vec → 100M chunks = 424 GB. That number is why quantization exists.
| Representation | Bytes/vec | Recall@10 |
|---|---|---|
| fp32 | 4,244 | 0.98 |
| fp16 | 2,196 | 0.98 (free, always do it) |
| int8 scalar | 1,172 | 0.97 |
| Binary + rescore top-200 | 276 | 0.71 raw / 0.96 rescored |
- Losing 4 pts of index recall ≈ 3.5 pts end to end (reranker Recall@5 ≈ 0.87 is the ceiling), buying a 340x latency cut. Below ~1% filter selectivity, partition the vectors; never post-filter (returns near-zero results and lies “no documents”).
Cost: money is not where you expect
- Ingest: parsing (PDF/OCR) ≈ 98% of the bill; embedding ≈ 1%. Fix = don’t OCR already-digital text, not a cheaper embedding model.
- Query: generation ≈ 89%; whole retrieval stack ≈ 11%; ANN itself ≈ 0.1%. Optimization order: context size (5→3 passages ≈ 16% off) → rerank vendor → never
efSearch. - Storage is the recurring bill (HNSW fp32 46.6 GB ≈ $687/mo w/ replicas); quantization is the 2nd-largest cost lever, behind only generation context size.
- Reranking = ~45 ms of a ~4.7 s response, the single largest quality lever. “Too slow” measures the wrong denominator.
- Prompt caching silently saves $0 if the prefix is below the vendor floor (
cache_creation_input_tokens= 0, no error); pad past it, put variable passages after the breakpoint.
Silent failures, each with its detector
| Failure | Detector | Control |
|---|---|---|
| Embedding version skew | age distribution of retrieved docs; p50 top-1 cosine | model id in index name + on every vector, checked per query; blue/green |
| Cross-tenant leak | canary tenant + sentinel strings, nightly | tenant scope in constructor; no tenant param on search |
| Orphan chunks | chunk count vs current version | deterministic ids + atomic replace-by-doc |
| Doc vanishes mid-update | present upstream, zero chunks | one atomic replace; else insert-first |
| Half-swapped alias | corpus_version assert at fusion | atomic multi-index flip; no swallowed KeyError |
| Backfill starves incremental | staleness p99 | separate queues/pools, rate-limit backfill embedding |
| Chunk-boundary loss | Recall@10 healthy, faithfulness low | structural splits; overlap; contextual chunking |
| Tombstone buildup | tombstone ratio/shard | compaction on the erasure SLA |
| Index degraded | nightly ANN-vs-flat recall audit | gate the alias flip on it |
| Stale answer, valid citation | — | never key an answer cache on query_hash alone |
- Eval has 3 layers: retrieval (Recall@k, nDCG, MRR) → generation (faithfulness, relevance, citation accuracy) → system (ANN-vs-flat recall, staleness p99, model-id match, orphan/tombstone counts). Diagnose retrieval first; nobody builds the system layer, and it catches the failures with no other detector.