RAG for Agents

Agentic RAG is the agent loop with a retriever behind a tool schema: the model decides whether to search, so retrieval fires zero, one, or several times with refined queries.

Core vocabulary

• RAG: look the question up in your own docs, paste the passages into the prompt, answer from text the model can read.
• Retriever: takes a query string, returns a ranked list of scored, identified passages. Never writes prose.
• Corpus / chunk: the searched documents; each is cut at index time into passages of a few hundred tokens (a token ≈ 4 chars of English). Retrieval returns chunks, never whole docs.
• Index time (offline, once per doc version: cut, embed, store) vs query time (while the user waits, on the latency budget). Push work to index time.
• Embedding: fixed-length float vector (~1,024 dims) where related texts point the same direction. Two embedding models are not interchangeable.
• Pooling: collapses one-vector-per-token into one vector per chunk (usually averaging). Where information is lost; source of the embed and search failures.
• Top-k: keep the k highest-scoring chunks.

Cosine similarity

• Measures direction, ignores length: a one-sentence chunk and a full page can score identically.

cos(a,b) = (a·b) / (|a| · |b|)

• ~0.31 = noise floor (query vs random doc); ~0.89 = strong match. Illustrative anchors from one corpus/model, not constants; the floor moves with the model.

Classic vs. agentic

• Skipping is the whole game: forced retrieval spends attention (weights sum to 1, a fixed budget) on passages that dilute the question. A quality bug before a cost one.
• Multi-hop needs two lookups in sequence (query 2 unwritable until query 1 returns); one pass structurally cannot do it.

Cost mechanics

• Every search is re-sent on every later call (history is re-sent as input), so cost grows quadratically (n²) with number of searches while token count grows linearly.
• Reference workload at claude-sonnet-5 ($3/M in, $15/M out): classic = 1 call = $0.01665 (the denominator).

• “2–6×” band = two to four searches. Skipping (0.58×) beats classic because passage tokens never enter the prompt.

When to retrieve

• Rule: retrieve when the answer depends on a fact that is (a) specific to your corpus AND (b) able to change. Skip when either fails.
• Break-even at corpus-question rate p ≈ 0.58: below it, letting the model skip is cheaper and higher quality; above it you pay a premium for the option to abstain. If corpus questions need two searches, break-even falls to p ≈ 0.22 (you’re buying capability, not savings).
• Wrong skip: model answers from background knowledge instead of searching. Prevented by the tool description, so the boundary belongs in the prompt.
• Recall@5 with a reranker (slower second model reordering a shortlist) gets the gold passage (human-labelled) into the top 5 ~87% of the time.

The six-stage pipeline and its failures

Index-time path (chunk → embed → index) once per doc version; query-time path (rewrite → hybrid search → rerank → assemble → generate) on the latency budget.

Hybrid search + rerank (cheapest quality win)

• Dense finds paraphrase, misses rare literals; BM25 (lexical, weights rare words by inverse document frequency) finds rare literals, misses paraphrase. Inverse blind spots, so fuse them.
• RRF merges by rank, not score (cosine and BM25 scores are incomparable):

score(d) = Σ_i 1 / (k_rrf + rank_i(d))    k_rrf = 60

• k_rrf = 60 (from the 2009 paper) flattens the head so being decent in both lists beats being excellent in one; no penalty for absence. Unrelated to the k in top-k.
• Bi-encoder index embeds query and doc separately (cheap, precomputable); cross-encoder reranker runs attention across query + passage together (accurate, slow). Funnel: retrieve 50, rerank, keep 5.

Retrieval as a tool

• Loop: send conversation + tool schema; if stop_reason != "tool_use", return text; else run each search, append results as user-role tool_result blocks, repeat.
• Four deliberate choices: description says when NOT to search; passages carry [doc:id] inline (copyable citations); include scores; empty case returns “No matching documents”, not "".
• Cap steps (a tries counter / max_steps) so the loop can’t run forever; provide a give-up branch or the query drifts and the model answers from prior knowledge.
• Caching: mark stable prefix cache_control: ephemeral, but nothing caches below the model’s minimum cacheable length (opus-5: 512, sonnet-5: 1,024, haiku-4-5: 4,096 tokens). Verify via usage.cache_read_input_tokens.

Advanced variants

RAG vs. the alternatives

• Facts that change → RAG.
• Live/transactional data → tool call to the system of record. Vector indexes are approximate (~340× speedup, ~0.96 index recall), fine for ranking, disqualifying for “find order #88213”.
• Format, tone, task style → fine-tune (teaches behavior, not facts; does not fix hallucination).
• Small stable corpus (< ~100k tok) → just put it in context. Cost break-even ≈ 35k tokens; the ~100k line is an engineering-cost judgement (deletes chunker, index, embedding version, a class of bugs).
• Domain vocabulary → fine-tune + RAG. Ship RAG first.

Evaluating retrieval

• Split first: a bad answer is either a retrieval or a generation failure; fixes are unrelated.

• Diagnose in dependency order: Recall bounds everything downstream; no prompt change makes the model cite what it can’t see.
• Plausible-but-wrong chunk-boundary failure passes Recall (chunk was retrieved) and answer relevance; only faithfulness catches it, because the claim is supported by no passage in the assembled context.
• Two “recall” names: index recall (approximate vs exhaustive search) ≠ Recall@k (gold passage in top k). An index at 0.96 can sit next to Recall@5 of 0.40.
• Third layer nobody builds (no quality metric goes red): index recall audit, indexing lag at p99, embedding provenance assertions, orphan/tombstone counts.