RAG for Agents

Retrieval-augmented generation (RAG) means looking a question up in your own documents and pasting the passages you find into the model’s prompt. The model then builds its answer from text it can read directly, not from what it absorbed during training.

In this lesson, we’ll build up the agentic form of RAG: retrieval as a tool the model chooses to call, not a step that always fires. Four questions run through it: how one retrieval works end to end, what it costs, how each stage of the pipeline fails, and which questions deserve a lookup at all. By the end you’ll be able to trace one retrieval through all six pipeline stages, price an agentic loop against a classic one, and defend the choice to skip a lookup out loud in an interview.

What goes in and what comes out

A question goes in as ordinary text. A short ranked list of document passages comes back out, each with a score and an identifier. That is the entire input and output of a retrieval step.

The component that performs it is the retriever: it takes a query string, searches the documents, and returns that ranked list. It never writes prose, the model does that afterwards, from what the retriever handed it. The documents being searched are the corpus.

Here is one retrieval against a corpus of product documentation. The user’s raw sentence goes in; two scored, identified passages come back, best match first.

INPUT — the question, exactly as the user typed it
  "we're on tier 2 — how many requests can we send before we start
   getting throttled?"

OUTPUT — the passages the retriever hands back, best match first
  [doc:api-limits]  (score 0.84)
  Tier 2 accounts are limited to 1,000 requests per minute per API key.
  Bursts to 2,000 rpm are absorbed for up to 10 seconds.

  [doc:api-errors]  (score 0.42)
  Requests over the limit return HTTP 429 with a Retry-After header.

Those two passages are pasted into the prompt above the user’s question. The model writes its answer out of them and cites the identifiers it was shown. Every fact below appears verbatim in one of the two passages:

"Tier 2 is 1,000 requests per minute per API key, with bursts to 2,000 rpm
 absorbed for up to 10 seconds [doc:api-limits]. Past that you'll get an
 HTTP 429 with a Retry-After header [doc:api-errors]."

That is RAG in full. Everything else is detail on how those passages got scored and ranked, who decided a lookup was needed, and what it cost.

The key structural difference is that classic RAG retrieves once and then answers, for every question, while agentic RAG makes retrieval a tool the model calls when it decides it needs to: possibly several times with refined queries, possibly zero times. Almost every trade-off below comes from that difference.

The vocabulary, defined once

Two timing terms come first, because everything else is timed against them. Index time is the offline pass you run once per document version, before any user has asked anything: cut, embed, store. Query time is what happens while a user waits. Work you can move to index time is effectively free; work at query time sits on the latency budget.

Chunk. One passage of a document. Documents are too long to retrieve or read whole, so at index time each is cut into pieces of a few hundred tokens: a token being the unit models count text in, roughly four characters of English. That cutting is chunking. Retrieval returns chunks, never whole documents.

Embedding. A fixed-length list of floating-point numbers (1,024 of them for a typical model) produced by a model trained on pairs of related texts, so that related texts come out pointing in similar directions. The dimensions have no names; training fixes the geometry, not the axes. That is also why two embedding models are not interchangeable: each invents its own coordinate system (the GenAI system-design RAG chapter).

Vector search. Embed every chunk once at index time and store the vectors. At query time, embed the question the same way and return the chunks whose vectors point most nearly the same direction. The store that holds those vectors and answers direction queries is a vector index.

Cosine similarity. How “most nearly the same direction” is measured, the cosine of the angle between two vectors. Two pointing the same way score 1.0, two at right angles score 0, bigger means more alike. It ignores length entirely, so a one-sentence chunk and a full page can score identically.

Pooling. The embedding model emits one vector per token; pooling collapses that sequence into a single vector for the whole chunk, usually by averaging over positions. It is where information is lost, because a few hundred vectors become one. Two of the six pipeline failures below (embedding and search) trace to pooling; the other four do not.

Top-k. Return the k highest-scoring chunks and discard the rest.

Cosine similarity, substituted

Cosine similarity is the dot product of two vectors (multiply matching coordinates and add) divided by both of their lengths, or norms:

cos(a,b) = (a·b) / (|a| · |b|)

Dividing by both lengths is exactly what removes length from the answer. Score a query against one on-topic and one off-topic document:

query  a = [0.8, 0.5, 0.1]   "annual plan refund window"
doc 1  b = [0.7, 0.6, 0.2]   "Annual plans are refundable ..."   on-topic
doc 2  c = [0.1, 0.2, 0.9]   "Invoices are issued monthly."      off-topic

cos(a,b) = 0.88 / (0.949 · 0.943) = 0.98
cos(a,c) = 0.27 / (0.949 · 0.927) = 0.31

That gap, 0.98 against 0.31, is the whole of what vector search runs on. And doubling doc 1’s vector doubles the dot product on top and the norm underneath, so the two cancel and the score stays 0.98: a one-sentence chunk and a five-paragraph chunk are on equal footing.

Two reference points make a score legible:

• ~0.31 is the noise floor. A query scored against a random document averages about that, so 0.31 is roughly what “no relationship” looks like.
• ~0.89 is a strong match, the best observed in one production corpus (the GenAI system-design RAG chapter).

Treat both as illustrative anchors from one corpus and one embedding model, not constants, the floor in particular moves with the model. To measure your own, score a few hundred pairs you know to be unrelated and take the mean.

What is inside the vector index, the structures that make direction queries fast and what they cost in memory, plus freshness, sharding, tenancy isolation, and the dollar cost model live in the GenAI system-design RAG chapter, which assumes this one. Every dollar figure below is sourced from there.

Classic vs. agentic

Classic RAG retrieves on every question. Agentic RAG lets the model decide. Here are the two designs side by side: one straight path against a loop with a bypass.

flowchart LR
    subgraph C["Classic RAG — fixed pipeline"]
        Q1([Query]) --> E1[Embed] --> S1[Search] --> P1[Stuff top-k] --> A1[Answer]
    end
    subgraph A["Agentic RAG — retrieval is a tool"]
        Q2([Query]) --> M((Model))
        M -->|search| R[Retriever]
        R --> M
        M -->|refine + search again| R
        M --> A2([Answer])
    end

The classic path is unconditional: embed, search, stuff top-k passages into the prompt, answer. One round trip, the same cost whatever was asked. In the agentic half the model sits between query and answer; the only way to reach the retriever is a tool call, and if the model emits none it takes the arrow straight to the answer. The refine + search again edge is the one classic RAG structurally cannot draw, because the refined query does not exist until the first search returns.

Six differences follow. The last row is the one this lesson is built around.

Skipping matters more than people expect. Classic RAG retrieves for “hi” and for “what’s 2+2”, and the injected passages do not sit in the prompt neutrally; they consume the model’s attention: the mechanism by which it weights each earlier token when producing the next. Those weights are normalized to sum to 1, which makes attention a fixed budget: every token of added text takes share away from everything already there, so forced retrieval is a quality bug before it is a cost one.

Multi-hop is the other structural difference. A multi-hop question needs two lookups in sequence, where the second query is only writable once the first has returned. “Which of our enterprise customers churned after the pricing change?” needs the pricing-change date first, then a churn query filtered by it. One pass structurally cannot do this. It would have to write both queries before either had run.

What the cost multiplier actually is

One billing fact drives everything here: every search an agent performs is re-sent on every call after it. The passages join the conversation history, and the whole history is re-sent as the input of the next call, so a search you paid for once is billed again at every following step.

Everything is priced against this reference workload (the GenAI system-design RAG chapter), at claude-sonnet-5 rates of $3 per million input tokens, $15 per million output:

Classic RAG makes one call (3,550 in, 400 out) for $0.01665. That is the denominator for every multiplier below.

Agentic RAG is not one call, so it is priced as the sum over every call the loop makes, not the last one. Each search costs a call to ask for it and another to use what came back. The prefix grows to ~1,200 tokens (the 1,050 above plus ~150 tokens of tool schema, the description of the search tool, present on every call), and each search adds a ~60-token tool_use request plus 2,500 passage tokens to the history permanently.

Cost grows faster than token count (3,760 to 6,320 tokens is 1.7×, but the cost doubles) because each search’s passages are re-billed on every following call. Across n calls the re-billing term grows with n² while the base grows with n, the same quadratic shape that governs any long agent conversation (LLM Internals). The commonly quoted “2–6×” figure for agentic RAG corresponds to two to four searches.

The top row is the point: skipping lands at 0.58×, cheaper than classic, because the 2,500 passage tokens never enter the prompt.

(The runnable listing later names claude-opus-5, priced at $5/$25 in Production and Cost, swap those rates in before costing that loop.)

The decision boundary, priced

The thesis of this lesson is knowing when not to retrieve. Here are both sides and the line between.

Retrieval earns its cost here. “What’s the rate limit on tier 2 of the API?” Nothing the model learned in training contains your tier-2 number, so it either reads it or invents one. Retrieval gets the gold passage: the one chunk a human labelled as containing the answer, into the final five about 87% of the time (Recall@5, measured with 50 candidates passed through a reranker, a slower second model that re-orders a shortlist, see the GenAI system-design RAG chapter). Searching once costs 1.31× against 0.58× to skip, so you spend 2.3× to go from roughly 0 to 0.87. Trivially worth it.

Retrieval actively hurts here. “Write me a Python function that reverses a linked list.” Classic RAG cannot abstain, so it searches. Your corpus has nothing relevant, which does not produce an empty result. It produces the five least irrelevant documents, scoring around 0.31, the noise floor. Three costs follow:

• Money: the five useless passages take the prompt from $0.00915 to $0.01665, 1.82× for zero information.
• Attention: the question falls from 4.76% of a 1,050-token prompt to 1.41% of a 3,550-token one, a 3.4× dilution of the only part that matters. Worse, retrieved passages sit in the middle of the window, and a model recalls facts worst in the middle (LLM Internals).
• Latency: roughly +73 ms of embed, search, fuse and rerank spent on nothing. Treat it as one pipeline’s stopwatch reading, dominated by the reranker; the durable point is the order of magnitude, tens of milliseconds on the user’s critical path.

The line between the two comes from the arithmetic. Let p be the fraction of traffic whose answer depends on a corpus fact. Always-retrieve pays $0.01665 regardless. Retrieval-as-a-tool pays $0.00960 when it skips and $0.02178 when it searches once (both carry the 150-token tool schema, an agent cannot call a tool it was never told about). Blending the two and solving for where the blend equals $0.01665 gives break-even at p ≈ 0.58:

Below roughly a 58% corpus-question rate, letting the model skip is cheaper and avoids the dilution. Past it you pay a premium for the option to abstain, and the quality argument has to carry it alone. The boundary is sensitive to one assumption: if corpus questions typically need two searches ($0.04164), break-even falls to about p = 0.22. On a multi-hop corpus you are buying capability, not savings, say so instead of pretending agentic RAG is a cost optimization.

The failure you accept in exchange is a wrong skip: the model deciding from its own background knowledge that it already knows your refund window. Preventing that is the job of the tool description below, which is why the boundary belongs in the prompt, not a separate router.

The rule, without the arithmetic: retrieve when the answer depends on a fact that is (a) specific to your corpus and (b) able to change. Skip when either fails.

• “What’s the rate limit on tier 2?”, satisfies both. Retrieve.
• “Reverse a linked list in Python.”, fails (a); no corpus of yours holds it. Skip.
• “What did we decide about pricing last week?”, fails (a) if the decision is a few turns up in this conversation, satisfies it if it was written to a document. That makes it a memory question first (Memory and Context) and a retrieval question second.

Classic RAG remains the right answer for high-volume, single-hop, latency-sensitive question answering over a stable corpus, where the round trips buy nothing.

The pipeline, and where it breaks

Every retrieval pipeline is the same six stages, and each fails in its own way. Two chains run through it and meet at hybrid search: one from Documents, at index time, and one from Query, while the user waits.

flowchart TD
    D[Documents] --> CH[Chunk]
    CH --> EM[Embed]
    EM --> IX[(Vector index)]
    Q([Query]) --> QR[Rewrite / expand]
    QR --> HY["Hybrid search: dense + BM25"]
    IX --> HY
    HY --> RR["Rerank: cross-encoder"]
    RR --> CX[Assemble context]
    CX --> GEN((Model)) --> ANS([Answer + citations])

The index-time path (chunk, embed, write to the index) runs once per document version; the index itself is a black box here (the GenAI system-design RAG chapter). The query-time path (rewrite, search, rerank, assemble, generate) runs once per question and is the only path on the latency budget, which is why the expensive work is pushed to index time.

Hybrid search and rerank give the highest quality gained per unit of effort spent. Most teams tune chunk size for a week and skip both.

The rewrite box resolves pronouns against the conversation (“does it support SSO?” → “does the Pro plan support SSO?”), adds vocabulary the corpus uses (“it crashed” → “crash, exception, stack trace”), and splits compound questions. In agentic RAG this box vanishes into the model, because writing the query is the tool call, which is why the agent later searches "rate limit tier 2" instead of the user’s whole sentence.

Three terms from the two high-value boxes:

• Hybrid search runs two search methods over the same query and merges their results.
• BM25 is a decades-old lexical formula: it ranks a document by how many of the query’s exact words it contains, weighting rare words far more heavily.
• A cross-encoder is the reranking model. It feeds the query and one candidate passage through the model together, so it can judge how well that passage answers that question. That is why it is accurate and slow: it runs once per candidate, cannot be precomputed, and only ever sees a shortlist cheaper search has narrowed.

The six stages and their characteristic failures are below, and the mechanism column is the load-bearing one; the fixes only make sense against it.

Three rows are routinely misread:

• Search. An identifier like ERR_4021 is cut by the tokenizer into several low-information fragments, and pooling then averages them into a mean over hundreds of tokens where they leave almost no trace. Vector search is not blind to the identifier; it simply cannot rank on it. That is the whole reason hybrid search exists.
• Rerank. Nothing in a top-k query asks for variety. If your corpus holds the same release note as a PDF, a help-center article, a changelog entry, and two translations, all five score near the query and fill k = 5, while the sixth-ranked passage that actually held the answer never enters the prompt. Dedupe by source document before truncating.
• Assemble. “Say how many were dropped” means putting a line like [3 additional passages omitted for length] in the prompt. A model that believes it has seen everything answers confidently from a partial view; a model told the view is partial hedges or asks to search again, and in a loop it can act on that. Silent truncation removes the only signal that would trigger the retry.

Chunking

Where a document gets cut decides what retrieval can ever find: a fact split across a boundary is a fact no chunk contains whole. Five ways to cut, the two in bold usually best:

Overlap, letting consecutive chunks share a few dozen tokens at the boundary, exists for one reason: so a fact split across a boundary appears whole in at least one chunk. It is compensation for bad boundaries, not a good in itself, so structural splits need much less of it.

What a bad boundary costs is clearest when you trace one to a wrong answer. The realistic case is a chat product several turns in, where earlier turns have eaten the window and only two passages fit, not five. A boundary error is survivable while the budget is generous and becomes an outright wrong answer once it tightens.

document "Billing FAQ", section 4, split at a fixed 500 tokens
  chunk 7 (ends):    "... Annual plans may be refunded within"
  chunk 8 (begins):  "30 days of the renewal date, minus any usage."

query: "how long do I have to get a refund on an annual plan?"

dense top-3
  1. billing-faq#7   0.71   has "annual", "refunded", "plans"
  2. billing-faq#2   0.64   general refund overview
  3. billing-faq#8   0.58   has "30 days", "renewal" — but not "refund"

context budget admits 2 passages   ->   chunk 8 is dropped

grounded model:      "Annual plans are refundable, though the specific
                      window is not stated in the documentation I have."
less-grounded model: "within 14 days."    <- fluent, confident, and wrong

The retriever did not fail. It ranked the chunk carrying the query’s vocabulary first, exactly as designed. The splitter created the failure at index time by cutting mid-sentence: the lexical signal (“annual”, “refunded”) stayed in chunk 7 while the content (“30 days”) went to chunk 8. The passage that scores well does not contain the answer, and the passage that contains the answer does not score well. Fifty tokens of overlap would put “may be refunded within 30 days” whole into chunk 7, and the failure disappears.

That trace also shows what two of the table rows buy. Sentence-window retrieves chunk 8 as a single sentence and hands the model its neighbours: the ranking unit stays small (precise scoring), the context unit grows (complete answer). Parent-document is the same trade one level up. Both decouple “what you match on” from “what you show the model.”

Keep this example for evaluation below: it is the canonical plausible-but-wrong retrieval, invisible to the metric most teams watch. Chunk 8 was retrieved, so Recall@10 scores it a success; faithfulness is what catches “14 days” being supported by no passage. The full fix table and per-option cost are in the GenAI system-design RAG chapter.

Contextual chunking comes with a caveat. It costs one small model call per chunk at ingest, about $2,200 per million documents against a $12,000 parsing bill, roughly 18% on top of what you already spend, so it is an easy yes to try. It works because a chunk reading “This limit was raised to 1000 in v3.” names no subject and is nearly unretrievable; prepending “From: API Rate Limits, Acme Platform v3 docs” gives the embedding the entities it needs to land near a query about Acme rate limits. You are repairing information that chunking destroyed. The “largest single-technique gain” claim comes from practitioner write-ups on particular corpora, not a controlled benchmark, so treat it as a strong prior. The gain scales with how context-dependent your chunks are: documentation full of “this limit” has much to repair; self-contained FAQ entries have almost none.

Hybrid search

Two search methods with opposite blind spots beat either alone, provided their result lists merge without comparing incomparable scores, and a reranker then narrows what the merge lets through.

flowchart LR
    Q([Query]) --> D["Dense: semantic"]
    Q --> B["BM25: lexical"]
    D --> F["Reciprocal rank fusion"]
    B --> F
    F --> R["Rerank top-50"] --> K["Top-5 to context"]

One query fans out to dense and BM25 search in parallel; RRF merges the two ranked lists; rerank scores the survivors properly; the top 5 are what the model sees. Fifty in, five out, the funnel is the point.

Dense is the vector search defined above (each chunk is a dense list of a thousand-odd floats). Lexical matches literal words. They have inverse failure modes, which is why fusing them works:

• Dense finds paraphrases and misses rare literals (pooling discards identifiers).
• BM25 finds rare literals because they are rare, and misses paraphrase. Its weighting term is inverse document frequency: a word in 3 documents of 100,000 counts as far stronger evidence than one in 40,000 of them.

So “how do I reset my password” is dense’s case and ERR_4021 is BM25’s. Run both and you cover both.

Reciprocal rank fusion (RRF). The two lists’ scores are not comparable, a cosine of 0.84 and a BM25 score of 34.2 are on unrelated scales. RRF throws the scores away and uses only ranks:

score(d)  =  Σ_i  1 / (k_rrf + rank_i(d))        k_rrf = 60

i runs over retrievers (one term per ranked list d appears in), rank_i(d) is where d placed in list i, and each placing contributes 1 / (60 + rank). The 60 is not derived from your corpus; it is the value from the paper that introduced the method (Cormack, Clarke and Buettcher, SIGIR 2009) and has been the default ever since. Note the name collision: this k_rrf has nothing to do with the k in top-k or Recall@k.

Three documents, their placings, and the sum:

B wins, and A, dense search’s first result, loses. That is the whole argument for fusion: being decent in both lists beats being excellent in one, because a document only one retriever likes is usually a document only one retriever’s failure mode liked. C shows the other side: being absent from BM25 costs it nothing punitive, RRF has no penalty, only accumulation, which makes it tolerant of a retriever that returns a short list or none.

The 60 flattens the head of each list: with it, rank 1 is worth only ~3% more than rank 3; with k_rrf = 0, rank 1 is worth 3× more and fusion collapses into “whoever’s top hit is loudest wins” (rerun the table at 0 and A beats B). The constant is what converts a rank into evidence, not a verdict, and it is why RRF is robust to one retriever being badly calibrated.

The reranker is the cheapest quality win because of a limit in the index. Your search index uses a bi-encoder: query and document are embedded separately and compared only as two finished vectors. That separation is the only reason an index can exist, every document vector is computed once, before any query arrives, but its price is that the query and document never see each other. The cross-encoder pays that back: attention runs across query and passage at once, so it can notice that this passage answers this question. It is far more accurate and far too slow to run over a whole corpus. Hence the funnel: retrieve 50 cheaply, rerank those 50 expensively, keep 5. Recall comes from the wide net, precision from the reranker.

Retrieval as a tool

The same loop appears at three levels of concreteness, then runs for one full turn as raw message traffic.

Tier 1 — Pseudocode

The one line to look at is if reply.done: the early return classic RAG does not have.

tools = [search_docs(query, filters), read_document(id)]
loop:
    reply = model(history, tools)
    if reply.done: return reply
    for call in reply.tool_calls:
        history += run(call)          # model decides IF and WHAT to search

The model now owns three decisions classic RAG makes for it: whether to search (it may return on the first pass), what query to use (it writes the string), and whether the results sufficed (the loop goes round and it may search again).

Tier 2 — LangGraph

A graph framework earns its place when the loop needs a retry: “grade the passages, and if they are bad rewrite the query and go back” is a state machine. This adds a grading node, the pattern is called Self-RAG, the model critiquing its own retrieval before using it.

LangGraph describes an agent as an explicit graph of nodes (functions) and edges (which node runs next). State is the dictionary passed between nodes; total=False makes every key optional, convenient, because each node returns only the keys it changed, and also what lets the seeding bug below exist.

from typing import TypedDict
from langgraph.graph import StateGraph, END


class State(TypedDict, total=False):
    query: str
    docs: list
    ok: bool
    tries: int          # MUST be seeded to 0 — an unset key makes give_up unreachable
    answer: str


def retrieve(s):  return {"docs": retriever.invoke(s["query"])}
def grade(s):     return {"ok": judge.invoke({"q": s["query"], "docs": s["docs"]})}
def rewrite(s):   return {"query": llm.invoke(f"Rewrite for search: {s['query']}").content,
                          "tries": s["tries"] + 1}
def generate(s):  return {"answer": llm.invoke(prompt(s)).content}
def give_up(s):   return {"answer": "I could not find this in the documentation."}


def route(s):
    if s["ok"]:          return "generate"
    if s["tries"] >= 2:  return "give_up"    # never loop forever
    return "rewrite"


g = StateGraph(State)
for name, fn in [("retrieve", retrieve), ("grade", grade), ("rewrite", rewrite),
                 ("generate", generate), ("give_up", give_up)]:
    g.add_node(name, fn)

g.set_entry_point("retrieve")
g.add_edge("retrieve", "grade")
g.add_conditional_edges("grade", route, ["generate", "rewrite", "give_up"])
g.add_edge("rewrite", "retrieve")           # the loop
g.add_edge("generate", END)
g.add_edge("give_up", END)
app = g.compile()

# Seeding tries is the caller's job: total=False lets the key be absent, and
# rewrite() would then raise KeyError on s["tries"].
result = app.invoke({"query": "how long do I have to get a refund?", "tries": 0})
print(result["answer"])

Four names are yours to supply: retriever (a vector store), judge (a model asked to grade passages, Evaluation), llm, and prompt. Everything else is the graph: five nodes, one conditional edge, one cycle (rewrite → retrieve → grade → rewrite).

Two things people get wrong. The give_up branch is easy to omit, but without it a bad grade always sends the graph back to rewrite, the query drifts further from the user’s question with each pass, and the model eventually answers from prior knowledge, the exact failure RAG existed to prevent. The tries counter is the only thing between a cycle and an infinite one, so an unseeded tries either raises KeyError (loud, lucky) or, if defaulted to something truthy, keeps give_up from ever firing. Seeding it belongs to the caller, not any node.

Tier 3 — Anthropic SDK

The same loop against the real API, using Anthropic’s Python SDK (the client library wrapping the HTTP calls). A tool here is a JSON schema describing a function the model may ask you to run: the model emits a request, your code runs it, you send the result back as another message. The most important line is the tool description’s sentence beginning “Skip it for”.

import anthropic

client = anthropic.Anthropic()

SEARCH_TOOL = {
    "name": "search_docs",
    "description": (
        "Search internal product documentation. Call this whenever the answer "
        "depends on product behavior, pricing, limits, or policy — never answer "
        "those from memory, since docs change weekly. Skip it for greetings, "
        "general programming questions, or anything already in this conversation. "
        "If the first search misses, call again with different keywords."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string",
                      "description": "Keywords, not a sentence. e.g. 'rate limit tier 2'"},
            "product": {"type": "string", "enum": ["api", "dashboard", "billing"]},
        },
        "required": ["query"],
        "additionalProperties": False,
    },
    "strict": True,
}

SYSTEM = """You answer from retrieved documentation only.

Rules:
- Cite the source id after every factual claim, like [doc:api-limits].
- If the passages do not contain the answer, say so and suggest what to search next.
- Never fill a gap with prior knowledge about how such products usually work."""

def rag_agent(question: str, max_steps: int = 6) -> str:
    messages = [{"role": "user", "content": question}]
    for _ in range(max_steps):
        resp = client.messages.create(
            model="claude-opus-5",
            max_tokens=4096,
            system=[{"type": "text", "text": SYSTEM,
                     "cache_control": {"type": "ephemeral"}}],   # stable → cached
            tools=[SEARCH_TOOL],
            messages=messages,
        )
        if resp.stop_reason != "tool_use":
            return next(b.text for b in resp.content if b.type == "text")

        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for b in resp.content:
            if b.type != "tool_use":
                continue
            hits = retriever.search(**b.input)[:5]
            body = "\n\n".join(
                f"[doc:{h.id}] (score {h.score:.2f})\n{h.text}" for h in hits
            ) or "No matching documents."
            results.append({"type": "tool_result", "tool_use_id": b.id, "content": body})
        messages.append({"role": "user", "content": results})
    return "Could not find a grounded answer within the step budget."

The loop is four steps: send the conversation plus the tool schema; if the model stopped for any reason other than wanting a tool, return its text; otherwise run each requested search and append the results; go round again. Supply a keyword-overlap stub for retriever.search and it runs end to end; a production system swaps that body for the hybrid dense + BM25 retrieval above and changes nothing else.

Four deliberate choices:

1. The description says when not to search. Without that boundary, agentic RAG retrieves on “thanks!”, the exact failure it exists to avoid.
2. Passages carry [doc:id] inline, so citations are copyable, not invented. A model asked to cite without visible ids fabricates plausible ones.
3. Scores are included, so the model can weigh a 0.31 hit differently from a 0.89 one. Include the scale in the system prompt if you want this reliably; a bare float means nothing without one.
4. The empty case returns “No matching documents”, not "". The model needs to see that in order to change strategy; an empty string reads as a malformed result.

The caching marker does nothing here, and it is worth seeing why. Marking the system prompt "cache_control": {"type": "ephemeral"} tells the API to store the processed prefix so later calls in the conversation bill at roughly a tenth of the input rate (LLM Internals). But a prefix only caches once it clears the model’s minimum cacheable length, and these floors are not ordered by generation (Production and Cost):

This loop’s prefix (SEARCH_TOOL plus SYSTEM) is about 290 tokens, roughly 57% of the 512 floor. Below the floor nothing caches, there is no error, and cache_creation_input_tokens comes back 0. Keep the marker anyway, a production agent carries more tool schemas and refusal rules in the same stable block and crosses 512 easily, but verify: read usage.cache_read_input_tokens on the second call and treat a zero as “the prefix is too short, or something in it is changing,” not as “caching is on.”

One turn, end to end

The loop actually running, as the messages array sees it, the tier-2 question from the opening. The model’s reply contains two blocks, a sentence of prose and a structured tool request:

USER
  {"role": "user", "content": "we're on tier 2 — how many requests can we
                               send before we start getting throttled?"}

--- client.messages.create(...)  ->  stop_reason: "tool_use" ---

ASSISTANT
  {"role": "assistant", "content": [
     {"type": "text", "text": "Let me check the API docs."},
     {"type": "tool_use", "id": "toolu_01A", "name": "search_docs",
      "input": {"query": "rate limit tier 2", "product": "api"}}]}

stop_reason: "tool_use" is the API saying the model stopped mid-answer because it wants a tool run; your code is now responsible for running it. Note the input: the user wrote fifteen words of conversational English, and the model searched four keywords and set a filter, the rewrite box, executed by the model instead of a separate pipeline stage, bought by the schema’s “Keywords, not a sentence” line.

Your code runs the search and hands the passages back. The role on that message is user, which is required, not stylistic:

USER   (tool results are user-role blocks)
  {"role": "user", "content": [
     {"type": "tool_result", "tool_use_id": "toolu_01A", "content":
        "[doc:api-limits] (score 0.84)\n"
        "Tier 2 accounts are limited to 1,000 requests per minute per API key.\n"
        "Bursts to 2,000 rpm are absorbed for up to 10 seconds.\n\n"
        "[doc:api-errors] (score 0.42)\n"
        "Requests over the limit return HTTP 429 with a Retry-After header."}]}

--- client.messages.create(...)  ->  stop_reason: "end_turn" ---

ASSISTANT
  "Tier 2 is 1,000 requests per minute per API key, with bursts to 2,000 rpm
   absorbed for up to 10 seconds [doc:api-limits]. Past that you'll get an
   HTTP 429 with a Retry-After header [doc:api-errors]."

Three of the four choices are now visible: the citations were copied out of the tool result character for character; the 0.84 hit is stated flatly while the 0.42 hit (near the noise floor) carries only a secondary detail; and there were two model calls where classic RAG makes one, the searches = 1 row, $0.02178 against $0.01665.

The next message shows the skip case:

USER       {"role": "user", "content": "thanks, that's exactly what I needed"}

--- client.messages.create(...)  ->  stop_reason: "end_turn" ---

ASSISTANT  "Happy to help — shout if you start seeing 429s."

No tool_use block, one model call: the searches = 0 row, 0.58× the cost. Classic RAG cannot produce this trace at any price, because its retrieval step has no if.

The genuinely hard case is “What did we decide about the pricing change last week?”, ambiguous about where the record lives. If the decision is a few turns up in this conversation, searching is the harm case with extra steps; if it was written to a decision log, not searching means answering from a half-remembered summary. The tier-3 description resolves it in one clause, “or anything already in this conversation”, which imposes an order: conversation first, index second. The boundary is not “is this a factual question” but “which store owns this fact,” a memory-architecture question before a retrieval one (Memory and Context).

Advanced variants

The pipeline and the loop above are the base pattern; each named variant extends one piece of it.

HyDE (hypothetical document embeddings) works because of the shape of the embedding space: questions and answers occupy different regions of it. “How do I rotate my API key?” is syntactically nothing like “Navigate to Settings → Security and click Regenerate,” so embedding the question lands you in the question neighborhood, away from your documents. HyDE has the model invent a plausible answer, embeds that, and searches with it, answer-shaped text lands near answer-shaped documents. The invented answer need not be correct: a wrong guess about the exact menu path still lands in the right neighborhood, which is all the search needs.

Late chunking follows from pooling. Ordinary chunking splits first and embeds each chunk independently, so the chunk “This limit was raised to 1000 in v3.” pools into a vector with no idea what “this limit” refers to, the subject was two paragraphs up, across a boundary. Late chunking runs the encoder over the whole document first, so every per-token vector has attended to the whole document, then pools per chunk. Same boundaries, same number of vectors, but each now carries context, contextual chunking’s benefit obtained mechanically instead of through an extra model call per chunk. The price: an encoder whose context window fits your documents.

Why GraphRAG is worth understanding starts with the limit of top-k: top-k retrieval answers questions whose answer lives in some chunk. “What are the recurring themes across all 400 incident reports?” has an answer that lives in no chunk. It is a property of the collection, so you could return the perfect top 5 and still not have it. GraphRAG builds entity and relationship structure at index time and traverses it, making aggregate questions answerable. The cost is a heavy indexing pass and a staleness problem on corpora that change often.

RAG vs. tool vs. fine-tune vs. long context

Retrieval competes with three other ways of closing a knowledge gap, and the choice turns on what kind of gap the system actually has.

flowchart TD
    Q{What kind of gap?} -->|Facts that change| RAG[RAG]
    Q -->|Live/transactional data| TOOL[Tool call to the system of record]
    Q -->|Format, tone, task style| FT[Fine-tune]
    Q -->|Small stable corpus < ~100k tok| LC[Just put it in context]
    Q -->|Domain vocabulary| BOTH[Fine-tune + RAG]

The root node is the move. Asking “what kind of gap?” is not the same as asking “should we use RAG”, the second question is how people end up with a vector database full of order records.

Facts that change point to RAG: knowledge that moves and must be cited, the case the rest of this lesson is about.

Live or transactional data points to a tool call. Order status is a database query, not a retrieval problem. Indexing an orders table into a vector database goes wrong two ways: the copy is stale the moment a row updates, and an exact-lookup question gets answered with approximate matches. “Approximate” is literal, production vector indexes search only part of the space, trading exactness for a roughly 340× speedup, so a recall of 0.96 (96 of every 100 true nearest neighbours found) is normal, fine for ranking passages, and disqualifying for “find order #88213” (the GenAI system-design RAG chapter).

That word “recall” names two different quantities, flag it before it does damage:

An index can sit at 0.96 recall while Recall@5 is 0.40: retrieving the true nearest neighbours is no help when the nearest neighbours are the wrong passages.

Format, tone, and task style point to fine-tuning. Fine-tuning is further training of the model’s weights on your examples. It teaches behavior, not facts, and does not fix hallucination, a confidently wrong answer in the right house style is still wrong.

A small stable corpus is a case for just putting it in context. Long context beats RAG below roughly 100k tokens: no chunking, no index, no retrieval failures, and with prompt caching the repeated prefix bills at about 10% of the normal rate. But at 100k tokens the no-index option is still about 2.2× dearer per query ($0.03615 cached against classic RAG’s $0.01665), and the cost break-even is far lower, solving for where the cached total equals $0.01665 gives about 35,000 tokens. So the ~100k line is an engineering-cost judgement, not a token-cost one: below it, a couple of cents a query buys you the deletion of a chunker, an index, an embedding-model version to track, and a whole class of retrieval bug, cheap against the salary that would maintain them. Above it the per-query cost grows large enough that maintenance stops looking expensive, and the answer sits in the middle of the window, where the model is least reliable (LLM Internals). “Is the corpus small enough to cache in context?” is worth asking before building any retrieval stack.

Domain vocabulary calls for both fine-tuning and RAG, because the levers act on different things. Fine-tuning acts on comprehension: after it, a token like ERR_4021 or an industry abbreviation is a meaningful unit in this domain, not noise, and if you fine-tuned the embedding model too, the query lands in the right region. RAG acts on the fact: what ERR_4021 currently means changed last Tuesday, and no weight update tracks that. Ship RAG first regardless, hybrid search and reranking close most of the vocabulary gap far more cheaply, and fine-tuning an embedding model turns every stored vector into a versioned artifact you must later migrate (the GenAI system-design RAG chapter).

Evaluating retrieval

The first move in measuring is a split: a bad answer is either a retrieval failure or a generation failure, and the fixes are unrelated. The layer column tells you which stage a red number is blaming.

The three retrieval metrics are best read by their arithmetic, not their names. Take an evaluation set of 40 questions, each with one hand-labelled gold passage:

Recall@10 = questions whose gold passage appeared anywhere in the top 10
          = 16 / 40  =  0.40

MRR = mean of 1/(rank of the gold passage), scoring 0 when it never appeared.
      Three questions, gold at ranks 1, 3, and not in top 10:
      (1/1 + 1/3 + 0) / 3  =  0.44

nDCG@5 for the middle question (gold at rank 3):
      DCG  = 1 / log2(3 + 1) = 0.50      IDCG = 1 / log2(1 + 1) = 1.00
      nDCG = 0.50 / 1.00 = 0.50          (IDCG = the best possible ordering)

DCG (discounted cumulative gain) is the score a ranking earns once each relevant passage is discounted by how far down it sits; IDCG is the score the perfect ordering would earn; dividing one by the other puts every question on a 0-to-1 scale. What each metric asks differs:

• Recall@k asks did we get it at all: a yes/no per question, the only one that bounds everything downstream.
• MRR asks how far down: it collapses each question to its single best hit, right when one passage suffices and wrong when the answer needs three.
• nDCG is MRR’s general form: it discounts every relevant passage by 1/log2(rank+1), then normalizes, which makes a question with one relevant passage comparable to one with nine. (The logarithm punishes rank 3 less harshly than 1/rank does: 0.50 against 0.33 for the same ranking.)

Diagnose in order, because it is a dependency, not a preference. If Recall@10 is 0.4, the right passage is absent 60% of the time, and no prompt change can make the model cite what it cannot see; fixing generation first optimizes a downstream stage against a broken input (Evaluation). One caution about the split: the chunk-boundary failure above passes Recall@10 (chunk 8 was retrieved) and answer relevance, and only faithfulness catches it, because “14 days” is supported by no passage in the assembled context. Recall measures the retriever; faithfulness measures the whole pipeline, including the truncation you did after retrieving.

There is a third layer nobody builds, one neither table has and with no other detector: nothing in retrieval or generation quality goes red when it breaks. Four system metrics are worth naming:

• Index recall audit. Nightly, run a sample of queries through the approximate index and through an exact brute-force scan, and report what fraction of the exact top-k the index found. It degrades silently as the index grows and its structures fragment.
• Indexing lag at p99. The 99th percentile of how long a document takes to go from written to searchable. The average is useless; the failure you care about is the slow tail of documents stale for hours.
• Embedding provenance assertions. On the search path, assert that the stored vectors came from the embedding model you believe is in use. A version changing underneath an index raises no error; it quietly returns worse results forever (the GenAI system-design RAG chapter).
• Orphan and tombstone counts. An orphan is a vector whose source document no longer exists, so it can still be retrieved and cited after deletion. A tombstone is the marker recording a deletion; if the sweeper stops running they accumulate. Count both and alert on growth.

These live in the GenAI system-design RAG chapter.

Cheat sheet

One row per observable symptom, its mechanism, and the fix.

Reading this alongside the GenAI RAG chapter

The GenAI system-design RAG chapter assumes all of this one. Two places where they look like they disagree and do not.

Cost is the same saving against a different denominator. That chapter prices a query at $0.0187 and calls generation 89% of it, the single-pass serving path, one retrieval and one generation, the right denominator for a classic pipeline. The 0.58× to 6.27× band above is a multiplier on that generation line, not a competing figure: an agent running three searches pays the generation cost about four times over, so the chapter’s advice gets stronger here. (Cutting from five passages to three saves 1,000 input tokens, $0.003 per query, 16% of that chapter’s reference query, 18% of this one’s workload. Same saving, different denominator; use 18% inside this lesson.)

Caching raises the question of which cache the cheat sheet means. “Cache” in the sheet means the embedding and retrieval caches. The retrieval cache is keyed on index_version, so the thing that would make it wrong is the thing that invalidates it. It does not mean caching final answers, rated “High. Do not.” there, an answer cache keyed on the query alone goes stale while still emitting the citation that makes it look verified. The one safe variant keys the answer on the retrieved chunk ids plus their content hashes.

Two things that chapter adds, both code listings there, each one idea, neither visible in the agent-loop view:

• ScopedRetriever wraps the retriever to refuse a search unless the customer-scope filter and the embedding-model check are both applied, on the search path, because a check that runs once at construction never sees a vector.
• ingest() writes a document’s chunks and replaces all of them in one atomic operation. Deleting the old chunks and then inserting the new ones is the obvious, wrong implementation: it leaves the document absent from the index if the worker dies between the two steps.

Conclusion

Agentic RAG is not a new architecture. It is the agent loop with a retriever behind a tool schema, and everything follows from one change: the model decides whether to search, so retrieval can fire zero times, once, or several times with refined queries.

flowchart TD
    U([User question]) --> DEC{Answer depends on a<br/>corpus fact that changes?}
    DEC -->|no| ANS([Answer directly — 0.58x])
    DEC -->|yes| RW[Model writes keyword query]
    RW --> PIPE["Retrieve: hybrid dense + BM25 -> RRF -> rerank -> top-5"]
    PIPE --> GRD{Passages answer it?}
    GRD -->|no, retries left| RW
    GRD -->|no, out of retries| GIVEUP([Say it wasn't found])
    GRD -->|yes| GEN([Answer with citations])

The load-bearing points:

• Skipping is the whole game. Below roughly a 58% corpus-question rate, letting the model skip is both cheaper (0.58× versus classic’s 1×) and higher quality, because unneeded passages dilute the model’s fixed attention budget. Retrieve only when the answer depends on a fact that is specific to your corpus and able to change.
• Cost grows quadratically with the number of searches, because every search’s passages are re-billed on every following call. Two to four searches is the “2–6×” band.
• Chunking decides what retrieval can ever find. A fact split across a boundary is lost before search begins. Prefer structural chunking; use contextual chunking where chunks are meaningless alone.
• Hybrid search plus a reranker is the cheapest quality win: dense and BM25 have inverse blind spots, RRF merges them on rank, and the cross-encoder buys precision on a shortlist.
• Diagnose in dependency order. Recall bounds everything downstream; the plausible-but-wrong chunk-boundary failure is invisible to recall and only faithfulness catches it.

Further reading

• Cormack, Clarke & Buettcher, “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods,” SIGIR 2009, the source of RRF and its k = 60.
• Robertson & Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” 2009, the lexical scoring behind hybrid search.
• Gao et al., “Precise Zero-Shot Dense Retrieval without Relevance Labels,” 2022, the HyDE method.
• Asai et al., “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” 2023, grading and abstention.
• Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” 2023, the position curve behind the dilution argument.
• Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” 2024, the GraphRAG design.
• Anthropic, “Introducing Contextual Retrieval,” 2024, contextual chunking and its ingest cost.

Next: 06 — Multi-Agent.