01 — Agent Foundations

Most systems that get called agents are not agents. This chapter gives you a test that settles the question, and then builds the one loop that every real agent is made of.

By the end you should be able to:

• define the six words the rest of the series runs on — model, tool, loop, harness, agent, orchestrator;
• put any proposed system on one of three tiers, using two questions;
• write the agent loop from memory, in pseudocode and against the real API;
• read the exact message array that goes over the wire, turn by turn;
• name the five mistakes that produce the errors you will actually hit.

The sentence worth carrying into an interview is “that is a workflow, not an agent” — defended with arithmetic rather than taste.

The shape of the whole thing

Before any mechanism, fix what goes in and what comes out.

You hand the system a goal in plain English — “fix this failing test.” It hands back a final text answer, plus whatever its tools changed in the world along the way.

In between, the same kind of request is sent to the model over and over. Each request carries three things:

• tools — the definitions of the functions the model is allowed to ask for,
• system — the standing instructions,
• messages — the conversation so far.

Each response comes back as two things: a list of content blocks, and a single string called stop_reason that says why the model stopped.

What a content block is

A content block is one typed piece of a response: a small object with a type field naming what it is, plus the fields that type carries.

Two block types matter for this chapter:

• a text block (type: "text") holds prose the model wrote;
• a tool-use block (type: "tool_use") holds an id, the name of a tool, and the arguments to call it with.

One response can contain several blocks, of both kinds. Here is a two-block response — the model saying what it is about to do, then asking for it:

resp.content == [
    TextBlock(text="I'll look up the weather in Paris."),
    ToolUseBlock(id="toolu_01A", name="get_weather", input={"city": "Paris"}),
]

That is why the response is a list and not a string, and it is why the worked trace in A worked trace shows resp.content as an array.

Goal in; text plus side effects out. tools + system + messages up, content blocks + stop_reason back. That is the entire input–output contract. Everything else in this chapter is about the code that sits around it.

The six words, in plain language

Every one of these is used loosely in the wild. Here they mean exactly this:

• A large language model (LLM) — “the model” from here on — is a function from text to text. It has no memory between calls: whatever it needs to know must be inside the request you send. A token is the unit it reads and bills in, a chunk of text averaging about four characters of English.

• A tool is an ordinary function you wrote — get_weather, run_tests, send_email. You describe it to the model as a name, a sentence of prose, and a schema of its arguments (a machine-readable list of the argument names and their types). The model cannot execute anything. It emits a structured request saying “call get_weather with {"city": "Paris"}”, and your code decides whether and how to run it.

• The loop is the repetition that follows from that split. Call the model; see if it asked for a tool; run the tool; append the result to the conversation; call the model again with the longer conversation. It ends when the model stops asking for tools, or when your code stops it.

• The harness is all the code you write around the model — assembling requests, running tools, counting spend, deciding when to quit, writing logs. The model is one stateless function call; the harness is the actual program. Harness engineering is entirely about it.

• An agent is that loop plus one property: the model, not your code, chooses what happens next and when to stop. The three tiers makes that test precise.

• An orchestrator is a model call whose job is to split a task into pieces, hand each piece to a separate agent (a worker), and combine what comes back; chapter 06 builds one.

Two facts everything else rests on

These come from how models work.

1. The API is stateless. The API is the network endpoint you send each request to. Stateless means the server keeps nothing between calls — so turn 20 only knows about turn 1 because you resent turn 1.

2. You are billed per token, on every request. Resending a growing conversation on every turn is where an agent’s money goes.

Chapter 00 derives both from the mechanism; you do not need it to follow this chapter.

1. The three tiers

Most projects that call themselves agents should not be agents. Defending that claim about a real system takes more than taste: it takes a ladder of tiers, a test that places any system on a rung, and the arithmetic for what the top rung costs.

The ladder

Three rungs, ordered by how much freedom you hand the model. Read left to right: at each step, more of the control flow moves out of your code and into the model.

flowchart LR
    A["Single LLM call<br/>prompt → output<br/>rung 1 · least freedom"] --> B["Workflow<br/>you write the control flow<br/>rung 2 · one step up"]
    B --> C["Agent<br/>model writes the control flow<br/>rung 3 · most freedom"]

    style A fill:#2d6a4f,color:#fff
    style B fill:#40916c,color:#fff
    style C fill:#95d5b2,color:#000

Dark green marks the tier you should have to argue your way out of; each lighter shade is a step into more unpredictability. The same convention returns in Stop conditions.

What changes between the rungs is one substitution — who computes the next step:

• A single call has no step N+1 at all. One prompt, one output, done.
• In a workflow, you write the control flow. The sequence of steps exists in your repository, as code, before any request is sent.
• In an agent, the model writes the control flow at runtime, one decision at a time. It never exists anywhere you can read before the run.

Comparing the three tiers

Two terms the table uses:

• Fixed DAG — a directed acyclic graph: a set of stages wired together in advance, with no cycles. Every possible path through it can be drawn on a whiteboard.
• Trace — the recorded log of one run: every request sent, every response, every tool call and its result. Debugging an agent means reading traces, because there is no source code describing the path it took.

The cost column is in multiples of what a single model call costs. Why the cost gap is so large below derives the 5–20× rather than asserting it.

The dividing line is not tool use

A workflow can call tools. Tool use is not the test.

What distinguishes an agent is that the model chooses which step comes next, and when to stop. Two lines of pseudocode make the difference exact. f is whatever computes the next step, and a forward pass is one run of the model over its whole input, producing its next output:

workflow:  next_step = f(current_step)              # f is your code
agent:     next_step = model(history, tools)        # f is a forward pass

So in an agent, the function that picks the next step is not a function you can read. It is a model inference, and it runs afresh every turn.

That substitution is what buys you adaptivity and what costs you predictability. You cannot enumerate the paths, so you cannot test them exhaustively, so every guarantee has to come from invariants — properties you enforce on every path, such as “this agent can never send an email without approval.” That is chapter 07 in one sentence.

Running the rule on three real systems

A rule you have never applied is a rule you will misapply. So take three systems and run the substitution on each: one that reads like an agent and isn’t, one that genuinely sits on the line, and one that is unambiguously an agent. The middle one is what makes the definition usable, because a definition you can only apply to easy cases is a slogan.

(a) A model-based router — looks like an agent, isn’t.

A call to a small, cheap model — Claude Haiku — reads an incoming support ticket and picks one of four branches: refund, technical, abuse, fallback (Routing).

A model is choosing what runs next, so next_step = model(...) seems to fit. The verdict is still workflow, for two reasons.

First, the branch set is enumerated in your code at design time. So the reachable state space — the complete set of situations the system can ever be in, and therefore the complete set of runs you would have to exercise to test it exhaustively — is four paths, all of which you can test.

Second, the model makes exactly one decision per request. After the branch is chosen, the rest of the run is fixed code.

Chapter 02’s table is right to file Routing under “Who plans: You”: you planned all four routes, and the model only picked an index into your list.

(b) An evaluator–optimizer loop — the genuinely ambiguous one.

A generator drafts an answer, a separate model acting as a judge scores the draft, and on a failing verdict the loop runs again (Evaluatoroptimizer).

Now a model decides whether there is a step N+1. It is choosing when to stop, which is half of the agent definition.

The verdict is workflow with a bounded loop. The step sequence is still fixed — generate, then evaluate, then generate. The iteration count is capped by your code. And the model chooses from a two-element set: {iterate, stop}. You cannot enumerate every possible draft, but you can enumerate every possible path, which is what testability needs.

It flips, though. Let the judge return which of several repair tools to invoke rather than a pass/fail bit, or drop the round cap so the loop ends only when the judge says so, and both halves of the definition now hold: an unbounded action space and model-owned termination.

Same diagram, same two model calls per round, different tier. Which is why you classify by what the model is allowed to choose from, not by how the boxes are drawn.

(c) “Fix this failing test,” with a shell tool — an agent.

Try to write the paths down. The model may read the test, then the implementation, then a third file it inferred from the traceback (the list of function calls the failure printed on its way out). Or it may run the test first to see the failure. Or run git log on the file to find the regression.

It stops when the test passes, which nothing in your code can predict a turn ahead. There is no finite branch set to enumerate, because the argument space of a shell command is unbounded, and the number of steps is a property of the bug rather than of your design.

The verdict is agent. Note that the tool count here is one — which is why “does it call tools” was never the question.

The rule, sharpened

Here is what you actually say in an interview:

The model choosing among branches you wrote is a workflow. The model choosing from an unbounded action space — and choosing when to stop — is an agent.

Two diagnostics fall out of it, and you can check both before writing any code:

1. Can I draw the complete set of paths?
2. Does my code, not the model, decide when the run ends?

Two yeses mean you have a workflow.

Run them on the tier table’s examples: “classify a ticket” has one path, “extract → validate → route” has a fixed three, and “fix this failing test” has a number nobody can write down.

Why the cost gap is so large

An agent has no server-side memory, so every turn resends the full conversation so far. That single fact is the cost model.

Two symbols. Write P for the fixed prefix that rides on every request — the system prompt plus the tool schemas — and a for the tokens each turn appends on top of it.

turn 1 sends            =  P + 1·a
turn 2 sends            =  P + 2·a
turn n sends            =  P + n·a
                           ─────────────────────────────────────────
total over n turns      =  n·P  +  a·(1 + 2 + … + n)
                        =  n·P  +  a·n(n+1)/2      ≈   n·P + a·n²/2

The middle step is the whole derivation. Every turn pays P again, so the prefix contributes n·P. The per-turn additions stack up as 1a + 2a + … + na, which is a times the sum of the first n whole numbers — and that sum is n(n+1)/2.

Which convention: this chapter charges turn 1 its own a, giving n(n+1)/2. Deriving the numbers charges turn 1 no delta, giving n(n−1)/2, which is smaller by exactly a·n. Both round to a·n²/2 and both are defensible; the numbers here are internally consistent under this one. Never mix the two inside a single calculation.

Linear versus quadratic. A workflow with N independent stages sends N·(P + a), which grows linearly — double the stages, double the tokens. An agent’s total contains an n² term, so it grows quadratically — double the turns, and that term quadruples.

Now substitute real numbers: a 2,000-token fixed prefix (system prompt plus tool schemas), 500 tokens added per turn, 20 turns.

sum of 1…20      =  20·21/2                                        =      210
prefix cost      =  20 · 2,000                                     =   40,000
delta cost       =  500 · 210                                      =  105,000
agent, 20 turns  =  40,000 + 105,000                               =  145,000

ONE single call  =  2,000 + 500                                    =    2,500
20 single calls  =  20 · 2,500                                     =   50,000

145,000 / 2,500   =   58×    against one call
145,000 / 50,000  =  2.9×    against twenty calls

Always say which denominator you mean, because the two answers differ by more than an order of magnitude. A 20-turn agent costs 58× a single call, which is 2.9× what twenty single calls cost. The first number compares the agent to the cheapest thing that could have answered the question at all; the second compares it to doing twenty independent pieces of work.

The tier table above quotes the first kind. 5–20× a single call is the range you get from an agent of roughly 4 to 10 turns:

4 turns:   sum 1…4  = 4·5/2  = 10    →  4·2,000 + 500·10 = 8,000  + 5,000  = 13,000  →  13,000/2,500 = 5.2×
10 turns:  sum 1…10 = 10·11/2 = 55   → 10·2,000 + 500·55 = 20,000 + 27,500 = 47,500  →  47,500/2,500 =  19×

The multiple grows with every turn, which is why the range is open-ended. At 40 turns: sum 1…40 = 40·41/2 = 820, so 40·2,000 + 500·820 = 80,000 + 410,000 = 490,000. That is 490,000/2,500 = 196× a single call, and 490,000/100,000 = 4.9× forty single calls.

Keeping P in the formula is what keeps the number honest. Drop the prefix — set P = 0 — and the 20-turn total collapses to a·n(n+1)/2 = 210a, against 20a for twenty single calls. That is 10.5×, more than three times the 2.9× above, from the same agent, purely because the fixed prefix was dropped.

Which number you land on is entirely a question of how large your fixed prefix is relative to what each turn appends. That is why the first cost lever is making P cacheable, not making the agent shorter. (Caching means the provider stores the processed form of an unchanged prefix and bills roughly a tenth of the normal rate to reuse it; Prompt caching derived derives the rules, and The agent loop below shows where the prefix sits in a request.)

This is the honest reason to prefer a workflow, and it is a much stronger argument than “workflows are simpler.”

What interviewers probe: whether you reach for an agent reflexively. “This is a workflow, not an agent, and here’s the cost arithmetic” outscores any clever agent design.

2. The agent loop

Every agent is a variation on one loop, and almost everything that goes wrong with the loop goes wrong at its exits — the endings it must branch on, and the guards that end it when the model won’t.

Start with the picture. Every box prints a short identifier before its label — S for the user goal, M for the model call, D for the stop_reason branch, E for tool execution, A for the append, G for the guards, X for the halt, F for the final answer, P for the pause. The pseudocode in Tier 1 pseudocode labels its lines with the same letters, so you can read the two side by side.

The thing to look at is the diamond D. It has five outgoing arrows, and four of those five are branches people forget to write.

flowchart TD
    S(["S · User goal"]) --> M["M · Call model with<br/>history + tools"]
    M --> D{"D · stop_reason"}
    D -->|tool_use| E["E · Execute tools"]
    E --> A["A · Append results<br/>to history"]
    A --> G{"G · Guards pass?"}
    G -->|yes| M
    G -->|no| X(["X · Halt: budget /<br/>step cap / loop"])
    D -->|end_turn| F(["F · Final answer"])
    D -->|max_tokens| T(["T · Truncated — NOT success"])
    D -->|refusal| R(["R · Declined — content may be empty"])
    D -->|pause_turn| P["P · Paused — append<br/>and resend"]
    P --> M

    style S fill:#1d3557,color:#fff
    style F fill:#2d6a4f,color:#fff
    style X fill:#9d0208,color:#fff
    style T fill:#bc6c25,color:#fff
    style R fill:#bc6c25,color:#fff
    style P fill:#bc6c25,color:#fff

Four things to take from it.

S · User goal is the only input to the whole picture. Everything else in the diagram is state your harness carries between calls.

D · stop_reason is the string the model returns saying why it stopped. It has five branches because the table in Stop reasons you must handle lists five values you must handle. A loop coded from a four-branch diagram has the pause_turn bug built in.

Only two branches come back. tool_use goes round through execute → append → guards, and pause_turn goes straight back to the model call. The other three leave the diagram for good.

Three of the five branches are amber, and their labels say why they are dangerous.

• Truncated — NOT success (max_tokens) and Paused — append and resend (pause_turn) are both well-formed success responses — HTTP status 200, the code meaning the request itself succeeded — that a naive else: return text hands back to your caller as the finished answer.
• Declined — content may be empty (refusal) fails the other way. It crashes on resp.content[0], because there may be nothing at index 0.

Emptiness is not exclusive to refusal, though. Tier 3 anthropic sdk shows an end_turn arriving with no text block and taking down the success path the same way. So the habit to build is: read stop_reason before you index content, on every branch.

Why pause_turn exists

pause_turn is the only non-final branch besides tool_use, and the diagram sends its arrow straight back to Call model with history + tools — because a pause is a turn that has not finished, not a turn that ended.

It comes from a server tool: a tool the provider runs on its own infrastructure rather than in your harness. You never see a tool_use block for it and you never write code to execute it. Web search and code execution are the usual examples — you declare the tool in your request and the provider does the work server-side.

The provider is running its own little loop over there — search, then read, then search again — and that loop has an iteration cap. When it hits the cap before the model is done, you get back everything produced so far with stop_reason: "pause_turn".

To resume: append that content to the conversation as the assistant turn, and send the request again with no new user message. The provider recognises the unfinished turn and picks up where it left off.

Adding a “please continue” message instead is the mistake. It changes the conversation the model is resuming.

The guard node, and why it sits where it does

A guard is a check your harness runs between turns that can end the run regardless of what the model wants.

Two nodes deserve their own treatment — Guards pass? (G) and Halt (X) — because their placement is a real design decision.

Guards pass? runs after results are appended to history and before the next model call. That is the only point in the cycle where the run’s full cost so far is known and no further money has been committed. Put it before tool execution and you halt on a budget you have not yet spent. Put it after the model call and you have already paid for the turn that broke the cap.

The Halt node names three causes, and each is a different piece of harness state:

• Step cap — an integer counter of how many times round the loop you have been. This is the cheapest guard, and the one whose absence is mistake #5 in The five mistakes with the errors they produce.
• Budget — a running ledger of tokens or dollars, summed across every call in the run. A step counter cannot substitute for it, because turn cost is not constant: the quadratic above means turn 20 costs far more than turn 2.
• Loop — a repeated fingerprint of (tool, args), meaning the model has asked for the identical call it already made. Stop conditions defines the detector.

All three are the harness’s job, not the model’s, for the reason chapter 07 makes structural: a model asked to respect a budget it cannot observe is being asked to guess.

What actually happens on one iteration

Six things happen between one model call and the next. Steps 2 and 3 name the two phases of inference, which is where the loop’s cost and latency come from; The kv cache the most important mechanism in this chapter derives them.

1. Serialize. Your harness turns tools, system, and messages into one token sequence — in that order. That order is why stable content must come first if it is going to be cacheable.

2. Prefill. The whole input sequence is processed in parallel, in one pass. Processing a token produces a small bundle of numbers the model needs again for every later token, and the provider keeps those bundles in a KV cache (short for key–value cache — the two arrays that make up each bundle). If the opening span of your request matches a span the provider already processed, it reuses that stored cache instead of recomputing it, and bills you roughly 10% of the normal rate for the reused part.

3. Decode. Output tokens are generated strictly one at a time, each one re-reading everything before it. This is why output tokens cost several times what input tokens cost, and why a long answer takes noticeably longer than a long question. It is also why streaming exists — delivering each token as it is produced instead of waiting for the whole response — which is the remedy recommended in Stop reasons you must handle.

4. Tool request, if any. If the model wants a tool, it emits a structured tool-call block rather than prose. The provider’s decoder — the code on their side that turns the model’s output into the actual next token — forces that block to match your schema by refusing any token that would break it (Structured output is a guarantee not a request). The model does not enforce the schema; the code around it does, which is why the guarantee holds. stop_reason comes back as "tool_use".

5. Execute and append. Your harness runs the requested calls — the diagram’s Execute tools node — and then appends the results to history as one single user message. Mistake #3 in The five mistakes with the errors they produce is what happens when you send them as several.

6. Guard. Guards run on the now-longer history. If they pass, control returns to step 1 — the same Call model with history + tools node, but with a longer sequence. If they do not, the run halts.

Step 6 is the whole cost story. Nothing is remembered server-side; the only reason turn 20 knows about turn 1 is that you resent it.

And note that the Final answer node is reached only from end_turn. Every other exit from the diagram is a partial result wearing a success-shaped response object.

The four moving parts

Once the loop is in place, there are exactly four things left to design, and each has its own chapter:

• Context — everything the model sees this turn. Chapter 04.
• Tools — the action space, meaning the complete set of things the agent is able to do. Chapter 03.
• Stop condition — how the loop ends when things go right. Stop conditions below.
• Guards — what stops it when the stop condition doesn’t fire. Chapter 07.

3. Building it: three tiers

Here is the same loop written three times — as six lines of pseudocode, as an explicit graph in a framework, and as production code against the real API. Reading them in order shows you which lines are essential, which are library convenience, and which are the ones that break in production.

Tier 1 — Pseudocode

Six lines, one per node of the diagram in The agent loop. Read them side by side with it; the trailing comments name the node each line corresponds to.

history = [user_goal]
while True:
    reply = model(history, tools)                     # M: call model
    if reply.wants_no_tools: return reply.text        # D -> end_turn -> F
    results = [run(call) for call in reply.tool_calls] # E: execute tools
    history += [reply, results]                       # A: append, ONE message
    if over_budget() or over_steps(): halt()          # G -> X

One thing to read past first. reply.wants_no_tools collapses all four non-tool_use endings into a single branch. That is deliberate: this tier is the shape of the loop, not a loop you would ship, and the shape is “tools, or done.” Tier 3 below splits that one branch into the four the dispatch table in Stop reasons you must handle requires, and the difference between these six lines and those is almost entirely that split.

Two further details are choices rather than accidents.

history += [reply, results] appends reply whole — the assistant turn including its tool-call blocks — and then all of results as a single message. Splitting that list into one message per result is mistake #3 in The five mistakes with the errors they produce, the one that costs 3× with no error to tell you.

The guard sits at the end of the loop body, after the append. That is the G placement argued for in The guard node and why it sits where it does. Moving it above model(...) would halt the run over spend that has not happened yet.

Tier 2 — LangGraph

LangGraph is an open-source library that expresses the loop as an explicit state machine — a set of named nodes plus rules for which node runs next — instead of a while statement.

Seven of its names appear in the code below. Read these first, or the block is noise:

• StateGraph — the builder itself, and the first symbol in the block. You construct it with the state schema the graph carries, then hang nodes and edges off it.
• MessagesState — a prebuilt state schema whose only field is a messages list with append semantics. A node returns the messages it adds, not the whole history.
• ToolNode — a prebuilt node that reads the tool calls off the last message, runs them, and returns the result messages.
• START and END — the graph’s sentinel nodes, marking entry and exit.
• .compile() — freezes the assembled builder into a runnable graph. Nothing executes until you call it, and this is where the checkpointer is attached.
• add_conditional_edges — attaches a Python function that returns the name of the next node. This is why should_continue below is the same branch as the stop_reason diamond in The agent loop, written as a test over the last message instead of as a field on the response.

Two names in the code are yours rather than the library’s, and the block assumes you defined them earlier:

• llm — a chat-model client. For the Anthropic models used in Tier 3, that is ChatAnthropic(model="claude-opus-5") from langchain_anthropic.
• tools — the list of Python functions you want the model to be able to call, each carrying a name, a docstring, and typed arguments that LangGraph turns into the schema. llm.bind_tools(tools) attaches that list to the client, so every call it makes advertises those tools.

The value of the framework is not the abstraction. It is the checkpointer: the component that writes the graph’s state out after every node, so a run can be picked up again later. That is what buys you durable state, resumption after a crash, and human approval pauses that survive a process restart.

The block below builds the graph, wires the cycle, and runs it once:

from langgraph.graph import StateGraph, START, END, MessagesState
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver

def call_model(state: MessagesState):
    return {"messages": [llm.bind_tools(tools).invoke(state["messages"])]}

def should_continue(state: MessagesState):
    return "tools" if state["messages"][-1].tool_calls else END

g = StateGraph(MessagesState)
g.add_node("agent", call_model)
g.add_node("tools", ToolNode(tools))
g.add_edge(START, "agent")
g.add_conditional_edges("agent", should_continue, ["tools", END])
g.add_edge("tools", "agent")          # the cycle

app = g.compile(checkpointer=MemorySaver())
app.invoke({"messages": [("user", goal)]},
           config={"configurable": {"thread_id": "run-1"}})

The thread_id is the resume key: it names the conversation whose saved state should be loaded. Without a checkpointer at all, a crash at minute 39 of a 40-minute run loses everything.

But MemorySaver does not deliver that. It is an in-process, in-memory checkpointer. It survives a graph interrupt within one running process — which is what makes human approval pauses work in a notebook — and it dies with the process.

The crash survival promised above needs a durable backend that writes to disk or a database: SqliteSaver for a single host, PostgresSaver for anything running on more than one. MemorySaver appears in the block above because it imports without a database behind it. Treat it as the demo, not the design.

Tier 3 — Anthropic SDK

This is the real thing: the loop written directly against the API through the official Python client library (its software development kit, or SDK), with no framework in between. It is short enough to write from memory, and worth being able to.

Read it top to bottom. The part that matters is the run of stop_reason checks before the tool-handling code — five branches in a deliberate order, which the notes after the block unpack one at a time.

import anthropic

client = anthropic.Anthropic()

SYSTEM = "You are a weather assistant. Answer only from tool results."

def handle_refusal(details) -> str:
    # stop_details is populated only when stop_reason == "refusal".
    cat = getattr(details, "category", None)
    return f"Declined ({cat}); nothing to retry with the same prompt."

TOOLS = [{
    "name": "get_weather",
    "description": "Get current weather for a city. Use when the user asks "
                   "about weather, temperature, or conditions right now.",
    "input_schema": {
        "type": "object",
        "properties": {"city": {"type": "string", "description": "City name"}},
        "required": ["city"],
    },
}]

def run_tool(name: str, args: dict) -> str:
    if name == "get_weather":
        return f"18C and raining in {args['city']}"
    raise ValueError(name)

def agent(goal: str, max_steps: int = 12) -> str:
    messages = [{"role": "user", "content": goal}]

    for _ in range(max_steps):                       # guard: step cap
        resp = client.messages.create(
            model="claude-opus-5",
            max_tokens=8192,
            system=[{"type": "text", "text": SYSTEM,
                     "cache_control": {"type": "ephemeral"}}],   # stable prefix
            tools=TOOLS,
            messages=messages,
        )

        if resp.stop_reason == "refusal":            # HTTP 200, content may be []
            return handle_refusal(resp.stop_details)
        if resp.stop_reason == "max_tokens":         # silently truncated
            raise RuntimeError("output truncated; raise max_tokens or stream")
        if resp.stop_reason == "pause_turn":         # server-tool turn paused, not finished
            messages.append({"role": "assistant", "content": resp.content})
            continue                                 # re-send with no new user message
        if resp.stop_reason != "tool_use":           # end_turn
            # the "" default is load-bearing: end_turn can carry no text block
            return next((b.text for b in resp.content if b.type == "text"), "")

        messages.append({"role": "assistant", "content": resp.content})

        results = []
        for block in resp.content:
            if block.type != "tool_use":
                continue
            try:
                out, err = run_tool(block.name, block.input), False
            except Exception as e:
                out, err = f"Error: {e}", True        # errors go back as results
            results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": out,
                "is_error": err,
            })

        messages.append({"role": "user", "content": results})

    raise RuntimeError("step cap exceeded")

The cache marker, and why it does nothing here

One piece of syntax first. "cache_control": {"type": "ephemeral"} on the system block is the caching marker from Why the cost gap is so large. Ephemeral here just means short-lived: the provider keeps the processed form of everything up to that marker for a few minutes and bills roughly a tenth of the normal rate to reuse it, then drops it. It is not storage you manage or can read back — you mark a boundary, and reuse either happens or it doesn’t.

At this toy’s size the marker is a no-op, and that is worth seeing once rather than discovering on a bill.

A cached prefix has to clear the model’s minimum cacheable length — the smallest prefix the provider will bother storing. That floor is 512 tokens on claude-opus-5, and the floors are not ordered by generation: 1,024 on claude-sonnet-5, 4,096 on claude-haiku-4-5 (Prompt caching the highest leverage lever).

Now measure this prefix. It is tools plus system, which render in that order:

get_weather schema, as JSON   =  279 characters
SYSTEM string                 =   59 characters
                                 ────────────────
total                         =  338 characters
338 / 4 characters per token  ≈   85 tokens        (rule of thumb, ch 00 §1)
85 / 512                      ≈   0.17  →  about a sixth of the floor

Nowhere near it under any tokenization (Tokens is where the four-characters-per-token figure comes from).

Below the floor nothing caches, there is no error, and the only symptom is cache_creation_input_tokens: 0 in the response.

Keep the marker anyway. A real agent’s dozen-odd tool schemas run to a couple of thousand tokens and clear 512 comfortably — that is the case the P in Why the cost gap is so large’s arithmetic describes. But verify rather than assume: print usage.cache_read_input_tokens on the second call, and if it is zero you are paying full price for the prefix on every turn.

Six lines carry the whole lesson

And none of them is the API call itself.

1. The stop_reason checks are ordered, and the order is the point. refusal, max_tokens, and pause_turn are all tested before the != "tool_use" fallthrough. Invert them and each becomes “not tool_use”, so each falls into the end_turn branch, and a truncated half-answer gets returned to your caller as the finished product. That is the silent-failure mode the table in Stop reasons you must handle warns about, produced entirely by a few lines sitting in the wrong order.

2. The pause_turn branch appends and continues rather than returning. This is the diagram’s P → M arrow in code: append resp.content as the assistant turn, then loop, which re-sends the conversation with no new user message and lets the provider finish the turn it was in the middle of. Without this branch, != "tool_use" catches pause_turn too and the loop returns a partial answer as though it were the final one — the exact bug Stop reasons you must handle calls the sneakiest of the five.

This agent declares only get_weather, which runs in your harness, so it will never actually pause today. The branch is here because it costs three lines now and is invisible to debug later, on the day someone adds a server tool to TOOLS.

3. A pause spends a step, because the branch is written as a continue inside for _ in range(max_steps). The counter is a lap counter, not a progress counter, and a resumed turn is a lap. Three consecutive pauses against max_steps = 3 exhaust the cap having run no tools and produced no answer, and the loop then exits through raise RuntimeError("step cap exceeded") — a true statement and a misleading diagnosis. If you use server tools, either give pauses their own counter or size max_steps knowing that some laps buy nothing.

4. messages.append({"role": "assistant", "content": resp.content}) appends the block list, not the text. resp.content holds both the text block and the tool-call blocks. Keep only .text and the tool result you send next references a call the API has no record of — mistake #1 in The five mistakes with the errors they produce, with the exact error it produces.

5. The end_turn branch’s next(...) carries an explicit "" default, and that second argument is the point. A bare next(b.text for b in resp.content if b.type == "text") raises StopIteration the moment an end_turn response contains no text block — an empty content, or a response whose only blocks are of some other type.

That is the same class of bug the refusal row in Stop reasons you must handle warns about, arriving on the success path instead: HTTP 200, a well-formed response object, and an uncaught exception out of your happy path. The default converts it into an empty answer your caller can test for — something a goal predicate (Stop conditions) can catch, which a StopIteration escaping a generator expression is not.

6. The except clause returns a string instead of re-raising. The line out, err = f"Error: {e}", True turns a tool failure into a tool_result marked is_error: true, which the model reads and can act on. Raising instead ends the run over a fixable typo — mistake #4.

One thing the code does not do: only one guard is implemented here. The for _ in range(max_steps) is the step cap. The budget ledger and the loop detector from the halt node in The guard node and why it sits where it does are yours to add. A production harness has all three, and chapter 07 builds them.

4. A worked trace

Abstract descriptions hide the thing that actually bites: the exact shape of the message array. Here is a real run, two turns long, message by message — and then the five ways people corrupt that array, each paired with the error it produces.

The question is “What’s the weather where the Eiffel Tower is?”

Turn 1 request — one message. The system and tools ride on every call and are not part of this array:

messages = [
    {"role": "user", "content": "What's the weather where the Eiffel Tower is?"},
]

Turn 1 response — stop_reason: "tool_use". Note that the response contains both a text block and a tool-use block:

resp.content == [
    TextBlock(text="I'll look up the weather in Paris."),
    ToolUseBlock(id="toolu_01A", name="get_weather", input={"city": "Paris"}),
]

Turn 2 request — you append two messages, and the array is now three long. Watch the two annotated lines: the assistant message carries both blocks verbatim, and the tool_use_id in the result matches the id the model handed you.

messages = [
    {"role": "user",      "content": "What's the weather where the Eiffel Tower is?"},
    {"role": "assistant", "content": resp.content},          # BOTH blocks, verbatim
    {"role": "user",      "content": [
        {"type": "tool_result", "tool_use_id": "toolu_01A",   # id must match
         "content": "18C and raining in Paris"},
    ]},
]

Turn 2 response — stop_reason: "end_turn", and you are done. Two model calls, one tool call, three messages.

The five mistakes, with the errors they produce

1. Appending only the text.

messages.append({"role": "assistant", "content": resp.content[0].text})   # ✗

The tool-use block is gone, but your next message contains a tool_result referencing toolu_01A. The API sees a result for a call that never happened, and rejects the request with HTTP status 400, meaning a malformed request:

400 — tool_result block(s) provided when previous message does not contain any tool_use blocks

2. Missing a result for a parallel call. The model can request several tools in one turn. Change the question to “weather in Paris, Berlin, and Tokyo?” and the same turn comes back with three tool-use blocks in a single assistant message:

resp.content == [
    ToolUseBlock(id="toolu_01A", name="get_weather", input={"city": "Paris"}),
    ToolUseBlock(id="toolu_01B", name="get_weather", input={"city": "Berlin"}),
    ToolUseBlock(id="toolu_01C", name="get_weather", input={"city": "Tokyo"}),
]

Now suppose the Tokyo lookup throws an exception, your except block logs it and moves on, and you send back only two results:

400 — messages.2: tool_use ids were found without tool_result blocks immediately after: toolu_01C

Every tool-use block needs exactly one matching tool_result, even when the content of that result is an error string. The correct handling of that failure is {"type": "tool_result", "tool_use_id": "toolu_01C", "content": "Error: ...", "is_error": True} — the block is mandatory, and its content is where the failure goes.

3. Splitting results across messages.

for r in results:
    messages.append({"role": "user", "content": [r]})        # ✗

This one is worse than an error — it works. Nothing is rejected. But you have shown the model a conversation in which parallel calls were answered one at a time, and the model imitates the pattern it is shown: it stops emitting parallel calls. Your agent silently gets slower over the course of a session, and no error tells you.

The slowdown factor is exactly the fan-out you gave up. Fan-out is the number of tool calls the model issues in a single turn. Three cities issued in one assistant turn cost one model round-trip and one batch of tool time; the same three issued one per turn cost three round-trips and three sequential batches. So it is 3× here because the example has three cities. The general figure is the average number of tool calls the model used to emit per turn — a task that fanned out to eight would degrade eightfold.

4. Raising on tool failure. A ValueError propagates out of your loop and the run dies. But the model can usually fix its own bad input in one turn, and a returned message like "Error: city 'Pariss' not found. Did you mean 'Paris'?" is a better instruction than anything in your system prompt.

Retry infrastructure failures — timeouts, a service being down — in the harness; return semantic failures, the ones caused by a wrong argument, to the model.

5. No step cap. This is the single most expensive line of code you can fail to write, because without it a confused model loops until your budget or your patience runs out.

5. Stop conditions

A step cap is only the bluntest way for a run to end. A run can end six ways, and they are not interchangeable: how far you should trust the result depends entirely on which ending fired.

Here are the six, in decreasing order of “the model decided”:

flowchart TD
    L[Agent loop] --> N{Why did it stop?}
    N --> A["end_turn<br/>model is satisfied<br/>acceptable"]
    N --> B["Goal predicate<br/>tests pass, schema valid<br/>ideal"]
    N --> C["Step cap<br/>you under-scoped it"]
    N --> D["Token / dollar budget<br/>you under-scoped it"]
    N --> E["Loop detector<br/>a bug — report it"]
    N --> F["Human veto<br/>a person intervened"]

    style A fill:#40916c,color:#fff
    style B fill:#2d6a4f,color:#fff
    style C fill:#bc6c25,color:#fff
    style D fill:#bc6c25,color:#fff
    style E fill:#9d0208,color:#fff
    style F fill:#1d3557,color:#fff

Why did it stop? is the question a trace has to answer. The green shading is the same ladder as The three tiers: darker is the ending you should have to argue your way out of.

Reading the six leaves in rank order:

• A goal predicate — a test your own code runs to decide whether the goal was actually met — is the ideal, in dark green.
• end_turn, model satisfied, is merely acceptable: mid green, the same rung the workflow tier sat on.
• Step cap and token / dollar budget are amber and say you under-scoped it.
• Loop detector is red: a bug, to be detected and reported, never silently swallowed.
• Human veto is navy — a person intervened.

Two of the six are mechanisms rather than counters, so they need defining:

• A loop detector computes a fingerprint of (tool_name, arguments) on every step — a short string derived from the call, such that two identical calls give the identical string — and halts when the same fingerprint recurs N times, since a model repeating itself is stuck rather than working. The naive version catches only literal repeats; a read/write/read cycle and a run-tests/edit/run-tests stall both defeat it, which is why real detectors also fingerprint a snapshot of the world state and track how many steps have passed since anything measurably improved (Loop detector builds all three; Infinite loops derives why loops self-reinforce).

• A human veto is a blocking approval gate on a specific class of action — not on the agent as a whole. The agent runs freely until it proposes something irreversible, then stops and waits for a decision that is issued outside its context, so no amount of generated text can forge it (Human in the loop, and the propose-then-confirm mechanism in Irreversible actions).

Why end_turn is weak evidence

A model does not decide to finish. It produces one token at a time, and it produces each one by scoring every word-piece it could say next and picking from that spread of scores. One of the things it can pick is a special piece that means this response is over. end_turn is the API telling you that is the piece that came out.

So end_turn is a statement about which token the model chose, not about the state of the world. It correlates with task completion because of how the model was trained — responses that were genuinely finished tended to end there — but nothing enforces the correlation, and when it breaks, the failure is silent and confident.

Prefer a predicate your harness can check. Left column: what people settle for. Right column: the mechanical replacement.

That last row is the trap. An empty task queue with an unmet goal is a failure, and an agent that reports it as success is worse than one that crashes — a crash gets investigated.

Stop reasons you must handle

Read this as a dispatch table. Every row is a branch your loop needs; the third column is the bug you ship without that branch; the fourth is what the branch body should do.

Five rows, not five stop reasons in total. The API documents six. The sixth is stop_sequence, and it is absent here because it cannot occur unless you asked for it: it means the model emitted one of the strings you passed in stop_sequences, and if you never pass that parameter — this chapter’s loop does not — you will never see the value. If you do pass one, treat it as an end_turn that stopped where you told it to, and read stop_sequence on the response to learn which string fired.

Every row in the table above is something the provider can hand you unbidden, which is why those are the five your loop has to branch on.

The refusal row is worth one extra note: it comes from a separate safety model screening the exchange, not from the model declining in prose. That is why the refusal arrives as a stop_reason and can leave content empty rather than filled with an apology.

The last three rows are the ones people don’t code for, and all three fail quietly. pause_turn is the sneakiest: the response is well-formed, content is populated, and the only thing wrong with it is that the model was not finished.

6. Reactive vs. proactive

Everything so far assumed a person watching the run. Take the person away — an agent triggered by a schedule rather than by a message — and one failure catches every team the first time: a message delivered twice to an agent that produces something different each run.

A reactive agent runs because a user sent a message. A proactive agent runs because a clock, a webhook — a request another system makes into yours the moment something happens there — or a file watcher fired.

The table contrasts them on the four dimensions that change your design; the bottom-right cell is where all the new work lives.

A proactive agent is a distributed system with an LLM in it. It needs everything an ordinary scheduled job needs:

• idempotency keys — so that doing the same unit of work twice has the effect of doing it once;
• dead-letter handling — somewhere to park messages that keep failing, instead of retrying them forever;
• at-least-once delivery guarantees — the queue promises a message gets through, and tolerates delivering it twice to keep that promise;
• logging and alerting loud enough that a failure nobody was watching still gets noticed.

And then non-determinism on top, meaning the same input does not reliably produce the same output.

What that looks like on a real system

A nightly job wakes at 02:00, pulls unanswered support tickets off a queue, and for each one runs an agent that reads the ticket, searches the docs, drafts a reply, and sends it.

Nothing about that agent’s loop differs from The agent loop — same stop_reason branch, same guards. What differs is the trigger. A failure at 02:04 is discovered at 09:00 by a customer, and there is no user sitting there to notice that the agent answered the wrong ticket.

The specific hazard: at-least-once delivery meets a non-idempotent agent

The collision, step by step:

1. A worker takes ticket #8412, drafts a reply, and sends it.
2. The worker dies before acknowledging the message.
3. The queue’s visibility timeout — the window during which a handed-out message is hidden from other workers — expires with no acknowledgement.
4. The ticket is redelivered. A second worker runs the same agent.
5. The customer gets two emails.

The fix is the same as in any distributed system: an idempotency key, a stable identifier for one unit of work, checked against a ledger before the irreversible act, and written to that ledger before the act rather than after it.

Where the LLM changes the usual answer

The hard part in a non-deterministic system is deciding what the key is derived from.

Derive it from the input, never from the output. The same ticket run twice produces two different drafts — different wording, different length, possibly a different subject line. So a key computed from the generated email is a different key on every retry, and the ledger becomes a growing log of duplicates that never recognizes a repeat.

Compute it from the ticket id and the ticket fields put into one fixed normalized form instead, deliberately excluding anything that varies per delivery: received_at, attempt, queue_message_id.

And compute it in the harness rather than asking the model for one. A model-generated key is just a token sequence with no stability guarantee across runs, which defeats the entire purpose.

Case study 02 implements exactly this, ledger states included.

7. Harness engineering

The harness is the code around the model, and most agent quality lives in that code, not in the prompt. Every one of its responsibilities exists because no prompt instruction can substitute for it.

The diagram below expands one turn of the loop from The agent loop into its parts. Everything inside the box boundary is code you write; the three dotted arrows are state that no single step owns.

flowchart LR
    subgraph H["Harness — you own this"]
        P[Prompt assembly] --> API
        API[Model call] --> R[Response parsing]
        R --> AUTH{Authorize}
        AUTH --> T[Tool dispatch]
        T --> C[Context management]
        C --> P
        B[(Budget<br/>ledger)] -.-> API
        L[(Trace log)] -.-> API
        G[(Loop guard)] -.-> AUTH
    end
    M((Model)) <--> API

    style H fill:#f8f9fa,stroke:#333
    style M fill:#1d3557,color:#fff
    style AUTH fill:#2d6a4f,color:#fff

Follow the solid cycle once:

• prompt assembly builds the request in tools → system → messages order;
• the model call goes out;
• response parsing turns the returned block list into typed structures and reads stop_reason;
• authorize decides whether each requested call is permitted;
• tool dispatch runs the permitted ones;
• context management decides what survives into the next assembly.

Then control returns to prompt assembly, which is why the cycle closes rather than terminating.

The three dotted arrows are not decoration. The budget ledger and the trace log attach to the model call, because that is the only place tokens are spent and the only place there is anything to record.

The loop guard, though, attaches to Authorize rather than to the model call, and that placement is an argument: a loop is detected from the (tool, args) a model asked for, so you want to catch it at the point where a repeated request can still be refused — before the tool runs, not after you have paid for the turn that requested it.

Authorize is the only highlighted node, because it is the one row in the table below that is a hard boundary rather than an optimization. Everything else in the loop makes the agent cheaper, faster, or more debuggable; authorization is what makes a wrong decision not matter.

Each row below is a harness responsibility and the mechanical reason a prompt instruction cannot cover it:

A better model with a bad harness loses to a worse model with a good one. Concretely: a harness that lets tool output flood the context will hit quality collapse around turn 30 regardless of which model you use. The failure is positional — models retrieve facts reliably near the start and end of their input and unreliably from the middle (Why quality degrades in long contexts) — rather than being about intelligence.

What interviewers probe: “Your agent is worse in production than in the demo. Where do you look first?” Weak: prompt tuning. Strong: pull traces; check cache_read_input_tokens, the usage field counting how many input tokens were served from cache (a deploy may have changed one byte of the cached prefix and dropped it to zero), the stop_reason distribution (a rise in max_tokens means silent truncation), tool error rates, and whether the search index the agent retrieves from was rebuilt. Prompt tuning is what you do after the diff tells you where it diverged.

8. When not to build an agent

Whether a system deserves an agent at all comes down to four checks. Run all four; one “no” means drop a tier.

Running the checks on one system

Take a real pipeline: “Invoice arrives → extract fields → validate totals → route to an approver.”

Complexity — no. Every invoice takes the same four steps, and the variation is in layout, not in procedure. The extract stage handles that layout variation with templates: one small per-vendor rule set saying where on the page each field sits, matched by whatever the invoice looks like. It is fully specifiable, so you drop a tier.

Value — no. Price both sides in the same unit: multiples of one single call. The workflow is four single calls, so 4×. A roughly 10-turn agent doing the same job is 19×, from Why the cost gap is so large’s arithmetic. That is 19 / 4 = 4.75, a little under five times the spend for the same output. Drop a tier again.

Viability — yes. Extracting fields from documents is something models are reliably good at today.

Cost of error — yes. A wrong total is caught by the validation stage or by the human approver, and nothing has been paid out.

Two “no”s, and the first one is load-bearing: the answer is a workflow.

Notice which check failed, though, because it names the escape hatch. The failure was complexity, and complexity is a property of the input rather than of the task type. So the roughly 5% of invoices whose layout the templates cannot parse are exactly the population that fails check 1 on their own, and exactly the population an agent should get. The check that fails tells you where the agent belongs.

The disguised-workflow trap

The most common startup interview prompt describes a fixed four-step pipeline and asks you to “design an agent for it.” The correct answer is to refuse the framing:

“Steps one through four are the same for every input, so that’s a DAG. I’d build a workflow with one LLM call per stage — cheaper, individually testable, and each stage’s failure is attributable. Then an agent escape hatch for the ~5% of inputs the pipeline can’t classify, where reasoning over an unusual case is genuinely needed.”

Case study 08 works this through end to end and shows the pipeline coming out ~27× cheaper on model spend and more accurate — because a deterministic path has no opportunity to be creative.

Cheat sheet

Each row is a symptom you will actually observe, the mechanism in this chapter that produces it, and the fix. Two fixes below are named rather than explained here: offloading means keeping bulky content out of the conversation and passing the model a pointer to it, and compaction means replacing a run of old turns with a short summary of them.

Next: 02 — Design Patterns — the nine architectures, each with its cost derived rather than asserted.