Agent Design Patterns

In this lesson, we’ll walk the nine standard ways to wire large language model (LLM) calls together into a working system, from a fixed pipeline whose every step you wrote by hand to a loop that generates its own next task. By the end you’ll be able to pick the simplest wiring that does a given job and name what it will cost before you build it.

We describe each pattern the same way: its shape, the problem it solves, its failure mode, and its cost. The point of the catalogue is to let you pick the simplest wiring that does the job, and to know what it will cost before you build it.

What all nine have in common

First, fix the shape of the whole system: what goes in and what comes out.

All nine take the same input: a task written in ordinary text, such as “refund order 4021” or “find out why checkout returns a 500 error on empty carts”.

All nine produce the same output: one final answer in text, plus whatever side effects their tools performed on the way: a refund issued, a file edited.

A tool is a function you wrote and then described to the model, issue_refund(order_id), read_file(path), search(query). The model cannot run it. It emits a structured request to call one, your code executes that request, and you feed the result back to the model as more text. Tools are the only way a model touches anything outside its own output.

So neither the input nor the output tells the nine apart. What differs is the wiring in between: how many model calls happen, who decides how many, and how much of the accumulated conversation each call is allowed to see. A design pattern here is a control-flow diagram over model calls.

Four terms the rest of the chapter leans on

Every cost argument below is written in these four terms.

Token. The unit a model reads and bills in, roughly four characters of English, so a 500-word page is about 650 tokens.

Context. Everything sent along with one call: the system prompt (the standing instructions that set the model’s role and rules), the tool definitions, the conversation so far, and the new request.

Context window. The hard cap on how large that context may get, the maximum number of tokens a single call can hold, prompt and answer together. Exceeding it is an error, not a gradual degradation.

Stateless. The interface you call remembers nothing between calls, so a multi-step agent has to resend the entire conversation on every step. That last fact is why context size, and therefore cost, is the axis every pattern in this chapter is really trading against.

How to read the N× cost numbers

Every N× below is defined against two quantities:

• P: the fixed prefix. The system prompt plus the tool definitions. Paid in full on every single call, and it never grows.
• a: the per-turn delta. What one lap of a loop adds to the conversation: the model’s own reply plus the tool result it triggered.

1× is one plain LLM call on a prompt of size P. N× means the pattern reads and writes N times as many tokens as that one call would have, answering the same task.

N× is a token multiplier at a fixed model tier, not a dollar figure. Moving work onto a cheaper model changes the dollars underneath a fixed multiplier, which is why Routing reads ~1.09× in tokens but 0.53× in dollars once its label has moved most traffic onto a cheaper rung (Routing).

The growth formula

Because the interface is stateless, history is resent every turn, so an n-turn loop sends this many input tokens over the whole run:

n·P + a·n²/2

n·P is the prefix term: each of the n turns resends the fixed prefix P.

a·n²/2 is the history term: turn t carries the a tokens every earlier turn contributed, so the deltas accumulate as a·(1 + 2 + ⋯ + n), which is about a·n²/2. The important fact is the shape. This term grows with the square of the turn count, so doubling the turns roughly quadruples it. Whenever this chapter says “quadratic growth” it means exactly this term. (The rounding from the exact n(n+1)/2 is a few percent, fine for arguing about growth, not for quoting a bill.)

How each pattern below is laid out

Every pattern is presented in the same order: the shape in plain words, the diagram and how to read it, the mechanism, when to reach for it, the failure mode with a concrete trace, and then the cost. Where the cost is a clean function of the wiring, the load-bearing numbers are shown; where the multiplier depends mostly on how long you let the pattern run, the figure is a range observed in practice, and the section says which lever moves you within it.

Picking one

You can choose among the nine before you know any of them in detail, because the choice turns on properties of your task, not of the patterns. The decision tree below asks the questions that discriminate and glosses each pattern as it reaches it.

Read the tree top to bottom. Each diamond is a question about your task; each rectangle is a pattern you land on. The second line inside every rectangle names how much freedom that pattern hands the model, pure workflow, bounded agency, or true agent. The paragraphs after the diagram define each pattern as the tree reaches it.

flowchart TD
    S0{Distinct input categories<br/>wanting different handling?} -->|Yes| ROUTE["Router<br/>pure workflow"]
    ROUTE -.->|then pick a pattern<br/>per branch| Q
    S0 -->|No| Q{Is the task<br/>decomposable up front?}
    Q -->|Yes, fixed steps| CHAIN["Prompt Chaining<br/>pure workflow"]
    Q -->|Yes, independent steps| PAR["Parallelization<br/>pure workflow"]
    Q -->|Yes, but count varies| ORCH["Orchestrator–Worker<br/>bounded agency"]
    Q -->|No| DYN{Is output quality<br/>cheaply measurable?}
    DYN -->|Yes| EVAL["Evaluator–Optimizer<br/>bounded agency"]
    DYN -->|No| ENV{Must it observe<br/>before every decision?}
    ENV -->|Yes| REACT["ReAct<br/>true agent"]
    ENV -->|No, checkpoints suffice| PLAN["Plan-and-Execute<br/>true agent"]

    style CHAIN fill:#2d6a4f,color:#fff
    style PAR fill:#2d6a4f,color:#fff
    style ROUTE fill:#2d6a4f,color:#fff
    style ORCH fill:#40916c,color:#fff
    style EVAL fill:#40916c,color:#fff
    style REACT fill:#95d5b2,color:#000
    style PLAN fill:#95d5b2,color:#000

Question 1 — distinct input categories

The first question is whether you have distinct input categories that want different handling. A refund request and a stack trace want different prompts, different tools, and plausibly different models.

If yes, put a router in front: one cheap classification call that reads the input, labels it, and hands it to the branch built for that label (Routing).

Routing is not a rival to the other eight. That is what the dotted edge back into the tree means: you then pick a pattern per branch. A router feeding a RAG lane and a ReAct lane is one design. (RAG is retrieval-augmented generation: fetch the relevant documents first, then answer only from them, see RAG for agents.)

Question 2 — decomposable up front

If you have no distinct categories, the second question is whether the task is decomposable up front: can you say now what the pieces are? There are three flavours of yes, and the distinction between them is the whole left half of the tree.

Yes, fixed steps, where you can name every step at design time, is Prompt Chaining, a fixed pipeline in which each call’s output becomes the next call’s input (Prompt chaining).

Yes, independent steps, where the steps exist but none reads another’s output, is Parallelization, where you fan the work out across calls that run at the same time and merge their results (Parallelization).

Yes, but the count varies, is Orchestrator–Worker (Orchestrator–worker). You know the shape of the decomposition, but not how many pieces this particular input needs. So one manager call, the orchestrator, splits the job into however many pieces the input demands, and hands each piece to a worker that starts from a blank context. That varying count is precisely why a model, and not your code, has to do the splitting.

Question 3 — cheaply measurable output quality

If the task is not decomposable, ask whether you can score the answer with something cheap: a linter (a program that flags style and correctness problems in code), a test suite, a checklist of required sources.

If you can, reach for Evaluator–Optimizer (Evaluator–optimizer), a draft-and-critique loop in which one call writes, a second call judges the draft against written criteria, and the writer revises until it passes or the round cap is hit. It buys you iteration you can trust, because something other than a vibe decides when to stop.

Question 4 — observation frequency

If quality is not measurable either, the last question is not whether the pattern observes the environment. Both remaining options do. The question is how often.

ReAct, short for Reason + Act, re-reads the world between every step, because step 3 is unknowable until step 2 returns (React reason act).

Plan-and-Execute: write the whole plan first with a strong model, then run the steps with a cheap one (Plan and execute), takes the “no, checkpoints suffice” branch. It observes too, but only at fixed points: once before planning and once per step boundary.

The discriminator is observation frequency, not observation at all. This is a common mis-selection, because “it needs to look at things” is true of both.

Three tiers of freedom

The second line of every box names its tier. Here they are in order of how much the model gets to decide:

• Pure workflow: Prompt Chaining, Parallelization, Router. You wrote the control flow, and the model only fills in the individual steps.

• Bounded agency: Orchestrator–Worker, Evaluator–Optimizer. The model gets one specific decision: how many workers to spawn, or whether to iterate again. The shape is still fixed by you, decompose, fan out, synthesize; generate, judge, revise, and so is the ceiling on how far it can go.

• True agent: ReAct, Plan-and-Execute. The model chooses what to do next, and nothing bounds that except a guard you remembered to add.

Start with a pure workflow, and escalate to the next tier up only when the simpler one demonstrably fails. Define “fails” concretely: a measured accuracy floor on a named input slice.

Why two patterns are missing from the tree

Reflexion and Autonomous Loop are absent on purpose. The tree picks a base loop, and neither of those is one.

Reflexion (Reflexion), a reflection loop that writes down why an attempt failed and carries that note into the next attempt, is a modifier. It is Evaluator–Optimizer (or ReAct, or Orchestrator–Worker) plus a store of lessons that survives the attempt. So you choose a base first, then decide separately whether those lessons transfer.

Autonomous Loop (Autonomous loop) is what a true-agent pattern becomes when you delete the human and let the loop generate its own next task. That is a governance decision (reversibility, budget, a success signal a program can check without a model) not a decomposition one.

Putting either in the flowchart would hide the question that actually decides it.

Prompt Chaining

The simplest pattern in the chapter is a fixed pipeline of model calls. One check keeps it correct: a gate that compares each step against the original input, not against the step before it.

The shape, in plain words. A prompt chain is a fixed sequence of calls in which each call’s output becomes the next call’s input, with ordinary code between the steps deciding whether the run continues. You wrote the sequence at design time; the model only fills in the individual steps.

The problem it solves. One call asked to do four things at once does all four adequately and none well. Four calls each asked to do one thing are each easy to prompt, easy to test, and easy to price.

The diagram below shows a three-step chain. The key element is the branch out of the Gate.

flowchart LR
    I([Input]) --> S1[Step 1] --> G{Gate} --> S2[Step 2] --> S3[Step 3] --> O([Output])
    G -->|fail| X([Reject])

    style I fill:#1d3557,color:#fff
    style O fill:#2d6a4f,color:#fff
    style X fill:#9d0208,color:#fff

The Gate is a plain code check, not a model call. Passing it carries the work forward; failing it ends the run at Reject. That terminal matters: the chain is allowed to stop, which is better than continuing on a bad step.

Mechanism. Each call gets a fresh, small context containing only what that step needs.

This is the pattern’s main advantage. Because context does not accumulate, you never pay the quadratic term from the growth formula. And no step suffers positional degradation: the U-shaped recall curve where content in the middle of a long window is retrieved worst (Why quality degrades in long contexts). Step 3 is as sharp as step 1.

Here is the whole pattern as pseudocode. The first line is a definition, it names the object the failure mode below turns on, and the four lines under it are the chain proper.

spec    = the original request the chain was handed, as a structured object —
          spec.text is the prose, spec.declared_fields the field names it declares
draft   = llm(f"outline this spec: {spec.text}")
if not valid(draft, spec): reject()
code    = llm(f"implement: {draft}")
tests   = llm(f"write tests for: {code}")

Here valid is given both the draft and spec, and that second argument is the subject of the failure mode below.

Use when: the task decomposes into steps you can name at design time, translate and then check the localization, or outline then draft then polish, or extract then validate then route.

Failure mode, error compounding. Step 1 hallucinates a field name, that is, states something no source supports, writing user_id when the spec says customer_id. Steps 2 and 3 then build faithfully on top of it. The output is internally consistent and entirely wrong, and nothing in the chain can notice, because each step only sees its predecessor’s output and never the original source.

The fix is the gate, and gates must check against the original input, not the previous step. Below, spec.declared_fields is the set of field names the spec declares, and fields(draft) is a small helper that pulls the field names the draft actually used.

# ✗ checks self-consistency — passes on a compounded error
if valid_json(draft): ...

# ✓ checks against ground truth. `<=` between two Python sets is the subset
# test: True only when every element on the left is also on the right.
# So this passes only if every field the draft used was declared in the
# spec — one invented `user_id` and the gate rejects.
if fields(draft) <= spec.declared_fields: ...

valid_json asks whether the draft is well-formed. The subset test asks whether the draft is true of the source, which is the only question that catches a compounded error.

A failed gate rejects; it does not continue. That is what the Reject terminal in the diagram is for, and it is the edge most often omitted. A chain that logs a gate failure and proceeds to step 2 has reintroduced exactly the compounding it was built to stop.

If a step is worth another attempt, retry that step with the gate’s specific complaint appended to its prompt, and cap the retries. An uncapped retry inside a chain silently converts a predictable N× into an unbounded one.

Cost. N× for N steps, each on a small context. Fully predictable.

Latency. In a chain, latency is serial: N × TTFT + N × decode. TTFT is time-to-first-token, the wait before the first output token appears, dominated by prefill (the single pass the model makes over the whole input before it can emit anything). Decode is the one-token-at-a-time generation that follows. Both terms multiply by N because nothing overlaps: step 2’s prompt does not exist until step 1 has finished decoding.

A chain also runs every input through the same pipeline on the same model. The next pattern is what you add when the inputs are not alike.

Routing

Most of a support queue does not need your best model, but something has to decide which requests do. A router, one cheap call that classifies the input and dispatches it to a handler built for that class, is usually the first cost lever to pull, ahead of rewriting any prompt.

The shape, in plain words. A router is a small model call that reads the incoming request, returns one label from a short list you defined, and hands the request to the branch wired to that label. Each branch is a different design: its own prompt, its own tools, and possibly its own model size.

The problem it solves. A single prompt covering every kind of input has to hedge across all of them, and hedging costs both accuracy and money. You end up running your most expensive model on requests a cheap one would have answered perfectly.

Throughout this chapter, Haiku, Sonnet and Opus are the small, mid and large tiers of the Claude model family, cheapest and fastest through to most capable and most expensive. They are one ladder, and choosing a rung is the lever this section is about.

flowchart TD
    I([Input]) --> R{Router · Haiku}
    R -->|refund| A[Refund handler<br/>Haiku + tools]
    R -->|technical| B[Tech handler<br/>Opus + RAG]
    R -->|abuse| C[Escalate to human]
    R -->|unknown| D[Fallback: general handler]
    A --> O([Output])
    B --> O
    D --> O
    C --> H([Human queue])

    style I fill:#1d3557,color:#fff
    style R fill:#bc6c25,color:#fff
    style O fill:#2d6a4f,color:#fff
    style D fill:#40916c,color:#fff
    style H fill:#1d3557,color:#fff

The four branches are not four copies of the same thing.

• refund goes to a Haiku handler that has write tools and a narrow policy prompt.
• technical goes to an Opus handler with RAG.
• abuse gets no handler at all. It is an escalate to human edge into a human queue. The load-bearing detail is that this branch never reaches Output, so no model ever composes a reply to an abusive message.
• unknown is the fallback: the branch that exists so the router is never forced to pick the least-wrong of k classes for an input that belongs to none of them.

Mechanism. A router buys two independent wins. The first is specialization: each branch gets a prompt tuned for one job, instead of one prompt hedging across all of them. The second is model tiering: the router is a ~200-token classification on the cheapest model, and its label decides which price tier the expensive call runs at.

Use when: the input space splits into a small number of named classes that want different tools, not just different wording, and you can label a few hundred real inputs to check the split is real.

Failure mode: the silent misroute. The model routes a technical question to the refund branch. The refund handler has no retrieval tool, so it answers from prior knowledge, confidently and wrongly. Branches have no path back, so nothing corrects it.

Three guards:

1. Always include a fallback route. Never force a choice among k known classes.
2. Log the routing distribution. A class that never fires means your router or your taxonomy is broken. A class that fires 80% of the time means your taxonomy is too coarse.
3. Let downstream handlers escalate back. A handler that finds itself with the wrong tools should be able to say so, not improvise.

The router in code

Here is the whole pattern against the Anthropic SDK, with the handlers stubbed. Two lines carry the design. The constrained instruction (“exactly one word”) makes the classifier’s output cheap and machine-checkable, but a model can still return Refund., an apology, or an empty string, so the label not in LABELS check is not optional: dispatching on an unvalidated string is a KeyError in production, and mapping anything unrecognized to unknown is the fallback branch doing its job. The dispatch itself is a plain dictionary, because once the label exists there is nothing left for a model to decide.

import anthropic

client = anthropic.Anthropic()

LABELS = {"refund", "technical", "abuse", "unknown"}

HANDLERS = {                                # plain Python — no model dispatches
    "refund":    lambda msg: f"[refund handler, Haiku + tools] {msg}",
    "technical": lambda msg: f"[tech handler, Opus + RAG] {msg}",
    "abuse":     lambda msg: "[human queue] escalated; no model composes a reply",
    "unknown":   lambda msg: f"[general fallback handler] {msg}",
}

def route(message: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",           # the cheapest rung does the classifying
        max_tokens=10,
        system="Classify the support message. Reply with exactly one word: "
               "refund, technical, abuse, or unknown. No punctuation.",
        messages=[{"role": "user", "content": message}],
    )
    text = next((b.text for b in resp.content if b.type == "text"), "")
    label = text.strip().lower()
    if label not in LABELS:                 # never dispatch on an unvalidated string
        label = "unknown"
    handler = HANDLERS[label]
    return handler(message)

print(route("I was charged twice for order 4021"))

Cost. With traffic shares sᵢ routed to models costing cᵢ, the bill is c_router + Σ sᵢ·cᵢ: the router’s own cost, plus, for each branch, the share of traffic that lands there times what that branch costs.

Prices are quoted per MTok, one million tokens, and always as a pair, input rate then output rate, because generated tokens cost several times more than read ones. This is the only price table in the chapter.

These are list prices as of mid-2026, so treat the date as part of the number. Vendors reprice and ship new tiers, so a figure quoted from memory a year from now will be wrong. The habit worth keeping is not the three rows but the ratios they encode: output is 5× input on every rung, and the small-to-large spread is also 5×. Those survive a price change; the absolute numbers do not.

Price one request. A 200-in / 5-out router call costs about $0.0002. A 2,000-in / 400-out handler costs $0.0040 on Haiku and $0.0200 on Opus. So the router is about 1% of the Opus call it gates, and the Haiku handler is 0.2× the Opus one.

Now blend the branches. With 60% of traffic answered on Haiku and 40% needing Opus, the bill is $0.0002 + 0.6·$0.0040 + 0.4·$0.0200 ≈ $0.011: against $0.0200 for sending everything to Opus, that is 0.53×, roughly half. Almost all of the saving is the tiering; the router term is a rounding error against it.

That gap between the router’s cost and the tiering it enables is why routing is usually the first cost lever, ahead of prompt optimization. The customer-support case study prices the same structure end to end by leave-one-out, turn off one optimization at a time and attribute to it whatever the bill rises by, and finds routing alone worth about a 2.8× reduction on the model bill, with caching edging it out at 3.1×. Routing still goes first, for a reason that is not about dollars: you cannot run 60% of traffic on Haiku until something has decided which 60%.

Parallelization

“Run it in parallel” names two different designs, one splits the work, the other splits the risk, and in both, the engineering sits in the combiner at the bottom of the diagram, not the fan-out at the top.

The shape, in plain words. Parallelization issues several model calls at the same time from one input and combines their answers into one. Two versions get conflated even though they solve different problems.

Sectioning splits the work: each call sees a different slice of the input, and the combiner stitches the slices back together.

Voting splits the risk: every call sees the same input, and the combiner compares the answers before deciding what to trust.

The diagram puts the two side by side. They have the same fan-out shape, so look at the labels on the boxes, not the arrows.

flowchart LR
    subgraph SEC["Sectioning — split the work"]
        I1([Input]) --> S1[Chunk A]
        I1 --> S2[Chunk B]
        I1 --> S3[Chunk C]
        S1 --> AGG1[Merge]
        S2 --> AGG1
        S3 --> AGG1
    end

    subgraph VOT["Voting — split the risk"]
        I2([Input]) --> V1[Lens 1]
        I2 --> V2[Lens 2]
        I2 --> V3[Lens 3]
        V1 --> AGG2[K-of-N]
        V2 --> AGG2
        V3 --> AGG2
    end

The two subgraphs differ only at the bottom: Merge stitches three different slices back together, while K-of-N compares three answers to the same input. The aggregator is where the difference actually lives.

Sectioning mechanism. Sectioning splits an input too large for one context, or a set of subtasks that never need to see each other. Each worker gets a small window, so, again, no quadratic term and no positional decay.

Voting mechanism, and the part people get wrong. Running the same prompt N times reduces variance: the run-to-run randomness that comes from the model sampling its next token instead of always taking the most likely one (Sampling and why temperature 0 isn’t deterministic).

But it does nothing about systematic error, the kind where the model is wrong the same way every time. If the model misreads the question, all N runs misread it identically. You get three confident wrong answers that agree, which reads as high confidence and is the opposite.

Variance is not the same as bias, the consistent lean in one wrong direction. To attack bias you need diverse lenses: N different prompts attacking the problem from different angles (correctness, security, performance) not N identical samples. Redundancy catches flakiness; diversity catches blind spots.

The aggregators are the hard part, not the fan-out

Merge is a design decision with a cost attached, and which design you pick depends on whether the chunk outputs overlap.

If they never overlap (findings listed per file, rows returned per database partition) merge is a concatenation: you join the pieces end to end in code, and it costs nothing.

If they overlap, contradict, or need to become one narrative, merge is another LLM call over all N outputs. That call is not included in the N× figure below, and it reintroduces a single long context at the end of a pattern you chose in order to avoid long contexts. Size it before you assume it is free: five 800-token chunk outputs is a 4,000-token merge prompt (trivial), but forty of them is 32,000 tokens, and the merge is now the most expensive call in the design.

K-of-N is a threshold you choose, not a property of the pattern. It says how many of the N lenses must agree before the system acts:

• 1-of-3 means “escalate if any lens objects”, catches a high share of real problems, raises a lot of false alarms, the right trade for a security gate.
• 3-of-3 means “ship only on unanimous pass”, almost everything it ships is genuinely fine, at the price of blocking constantly.
• 2-of-3 is the usual compromise.

The trap is what you do with the minority. Majority-voting a 2–1 split discards the dissent, and on diverse lenses the dissent is the entire product. The security lens objecting alone is not noise to be outvoted; it is the finding. So let K decide only whether to block, never whether to report. Surface every objection along with the lens that raised it. K-of-N as a silencer is defensible only in the redundancy case, where all N ran the same prompt and disagreement really is sampling noise.

Use when: reach for sectioning when the input exceeds the context window, or when the subtasks are genuinely independent. Reach for voting when a single sample’s run-to-run randomness is the risk you are buying down, and you can afford N× tokens to do it.

Failure mode: lost cross-chunk context (sectioning). A bug spanning two files is invisible to both workers, because neither sees the other’s file. Three mitigations: overlap the sections, give every worker a shared summary of the whole, or add a final pass that only looks at the boundaries.

Failure mode: correlated agreement (voting). N identical samples split sampling risk and nothing else. Three runs of the same prompt on a misread question return three matching wrong answers, and the aggregator reports 3-of-3, the highest-confidence output the system can emit, on its worst input. That is worse than a single call, which at least would not have claimed consensus. Detection: track the agreement rate per input slice. A lens set that agrees ~100% of the time is not three lenses, it is one lens run three times, and the fix is diversity in the prompts, not more samples.

Voting in code

The example below fans one diff across three lenses and combines the verdicts in plain Python, no fourth model call, because counting objections is arithmetic, not judgement. The PASS sentinel makes the combine step code: each lens either returns the exact string or an objection, so K-of-N reduces to a dictionary comprehension and a length check. The loop prints every objection before the block decision, K decides whether to block, never whether to report.

import anthropic
from concurrent.futures import ThreadPoolExecutor

client = anthropic.Anthropic()

LENSES = {
    "correctness": "Review the diff for logic errors only.",
    "security":    "Review the diff for security problems only.",
    "performance": "Review the diff for performance regressions only.",
}

def review(lens: str, instruction: str, diff: str) -> tuple[str, str]:
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=instruction + " If you find nothing, reply with exactly: PASS",
        messages=[{"role": "user", "content": diff}],
    )
    text = next((b.text for b in resp.content if b.type == "text"), "")
    return lens, text.strip()

diff = open("change.diff").read()
with ThreadPoolExecutor(max_workers=3) as pool:
    futures = [pool.submit(review, lens, inst, diff) for lens, inst in LENSES.items()]
    results = [f.result() for f in futures]

objections = {lens: text for lens, text in results if text != "PASS"}
for lens, text in objections.items():       # report every objection, always
    print(f"[{lens}] {text}")
print("BLOCKED" if objections else "PASS")  # K-of-N with K=1: any lens blocks

Swap the three lens prompts for three chunks of one long input and the same skeleton is sectioning, the only structural change is that the combine step becomes a concatenation instead of a threshold.

Cost. N× tokens, but wall-clock time stays ~1× because the calls run at the same time. Add the merge call on top if merge is itself an LLM call. This is the cheapest latency win in the catalogue, and the one case where cost and latency move in opposite directions, so quote both numbers.

Parallelization assumes you can enumerate the pieces yourself. The next pattern is what you build when only the model can.

Orchestrator–Worker

One manager call splits a task into however many pieces this particular input needs, each piece runs in a worker with its own blank context, and a final call combines what they return. The argument for it is capability, not speed.

The shape, in plain words. It is sectioning (Parallelization) with one thing changed: the model decides how many workers there are and what each one does. That is the line between a workflow and a model-directed pattern, but it is a bounded crossing, the model picks the fan-out, while you fixed the shape (decompose → workers → synthesize) and the ceiling on how many.

The problem it solves. Some tasks require reading far more material than fits in one conversation, and the only way to read it all is to read it in separate conversations that never meet.

flowchart TD
    I([Task]) --> O[Orchestrator<br/>decomposes]
    O --> W1["Worker 1<br/>own 60k window"]
    O --> W2["Worker 2<br/>own 60k window"]
    O --> W3["Worker N<br/>own 60k window"]
    W1 -->|~800-token summary| S[Synthesizer]
    W2 -->|~800-token summary| S
    W3 -->|~800-token summary| S
    S --> F([Result])

    style I fill:#1d3557,color:#fff
    style O fill:#40916c,color:#fff
    style F fill:#2d6a4f,color:#fff

Mechanism: context isolation, not parallelism. The orchestrator turns one task into w briefs, and it is the only call that sees the whole problem. Each worker then runs in its own window of ~60k tokens, its own conversation, its own tool results, starting from nothing but its brief. That isolation is why the workers cannot poison each other’s context, or the orchestrator’s. Each worker returns an ~800-token summary to the Synthesizer, which is a different call from the orchestrator so that it judges the returned evidence instead of defending the plan that produced it.

The 60k and ~800 are an illustrative configuration, not measured constants: 60k is a window a worker can fill with tool output without hitting the ceiling or the worst of the recall curve, and 800 tokens is about the length of a report that states findings with citations and nothing else. Both are yours to set. Their point is the ratio: six workers reading 60k each is 360k tokens explored, but the orchestrator only ever holds the six 800-token summaries, 4.8k tokens, a ~75× compression. A single agent physically cannot read 360k tokens of source and still have room to reason about it. That is the argument, and it is about capability, not speed.

Use when: the decomposition is real but how many pieces it has depends on the input, “audit every service that touches billing” when you don’t know how many that is, or “research this claim across whatever sources exist”. If you can write down the number of pieces at design time, use sectioning instead and keep the cheaper, more debuggable pattern.

Failure mode: worker conflict. Two workers edit the same file, and the second overwrites the first. Or two return contradictory findings and the synthesizer averages them into something true of neither:

worker_1: "The rate limit is 100 req/min [docs/api-v2]"
worker_2: "The rate limit is 1000 req/min [blog/scaling-2023]"
synthesis: "The rate limit is in the hundreds of requests per minute."   ← useless

Two fixes, both structural:

1. Disjoint scopes in the brief: no two workers may touch the same file or entity.
2. A synthesizer instructed to surface conflicts, along with where each finding came from, instead of reconciling them.

The fan-out in code

The whole structure fits in one page: an Opus call emits the decomposition as JSON, threads run the workers, an Opus call synthesizes. Three lines are load-bearing. json.loads(plan) is the contract check, if the orchestrator wraps its array in prose or code fences, fail loudly and re-ask instead of parsing harder. The [:6] cap is the “bounded” in bounded agency: the model picks the fan-out, your code holds the ceiling. And worker receives only its brief, that argument list is the context isolation the whole section argues for.

import anthropic, json
from concurrent.futures import ThreadPoolExecutor

client = anthropic.Anthropic()

def ask(model: str, system: str, user: str) -> str:
    resp = client.messages.create(model=model, max_tokens=1024, system=system,
                                  messages=[{"role": "user", "content": user}])
    return next((b.text for b in resp.content if b.type == "text"), "")

task = "Find the documented rate limits of our three upstream APIs."

plan = ask("claude-opus-5",                 # the one call that sees the whole problem
           'Decompose the task into independent research briefs. Reply with a '
           'JSON array of strings, e.g. ["...", "..."], and nothing else.',
           task)
briefs = json.loads(plan)[:6]               # contract check + a cap the model cannot move

def worker(brief: str) -> str:              # fresh context: the brief and nothing else
    return ask("claude-haiku-4-5",
               "Answer the brief in under 150 words. Findings and sources only.",
               brief)

with ThreadPoolExecutor(max_workers=len(briefs)) as pool:
    summaries = list(pool.map(worker, briefs))

print(ask("claude-opus-5",                  # a separate synthesis call, not the planner
          "Synthesize the worker reports. Surface conflicts with their sources; "
          "do not average them.",
          "\n\n".join(f"[worker {i}] {s}" for i, s in enumerate(summaries))))

Real workers are small tool-using loops, not single calls, this skeleton stubs them so the wiring stays visible.

Cost. The multiplier is (w·W + O) / B, built from four quantities you either choose or measure: w workers, W tokens each worker bills over its own short run, O the orchestrator’s own bill, and B what a single agent would have billed on the same question. The table substitutes two configurations at the ends of the usual range.

That is where the 4–15× range comes from: the high end has more workers reading deeper windows and is compared against a stingier single agent. (The multi-agent chapter derives the same two rows.)

Notice how small O is, about 6–10% of the bill. The orchestrator is not the expense; w × W is. So the levers that matter are worker count (linear), worker step budget (quadratic within each worker), and worker model tier, downgrading the orchestrator saves almost nothing. Moving the workers to Sonnet while keeping the orchestrator on Opus takes the whole design to about 0.64×, not “roughly half”, because the workers are ~90% of the tokens and Sonnet is 0.6× of Opus on both rates.

The multiplier measures work volume, not inefficiency. You are comparing one agent that stopped at 8 searches against 4 workers doing 8 each. Per unit of work, fan-out is cheaper, because w workers divide the quadratic term by w (what the 4–15× measures). Fan out to buy more reading, not to make reading cheaper.

Evaluator–Optimizer

A draft-and-critique loop sets two calls against each other: one generates, a second judges the result against criteria you wrote down, and the generator revises until the judge passes it or the round cap stops it. The load-bearing detail is that the judge runs as a separate call with a fresh context: it never sees the conversation that produced the draft.

The shape, in plain words. Generate, critique, revise, repeat.

The problem it solves. A first draft is usually close but wrong in specific, nameable ways. A model handed those specific complaints fixes them reliably, where the same model asked to “do better” simply produces a different first draft. The catch is that this only works when you can articulate the criteria, which is exactly the condition on the tree branch that led here.

The diagram has one loop and three exits. Two of them reach Output, and only one of those is a success.

flowchart LR
    I([Task]) --> G[Generator]
    G --> E{Evaluator<br/>fresh context}
    E -->|pass| O([Output])
    E -->|fail + structured feedback| G
    E -.->|max rounds| O

    style I fill:#1d3557,color:#fff
    style O fill:#2d6a4f,color:#fff
    style E fill:#bc6c25,color:#fff

Mechanism: why the fresh context matters. If you ask the same conversation “is this good?”, the draft is already in its context, and the model is now predicting the continuation of a conversation in which it just produced that draft. Two forces push it toward approval. Sycophancy is the model’s trained tendency to agree with whatever is in front of it. Positional recency is the fact that the most recent text in a context carries outsized weight on the next token. A separate call sees the draft as input to judge, not as something I just wrote. This is the same reason a code reviewer is not the author.

The edges out of the evaluator are the whole control flow. On pass, the draft leaves and the run is done. On fail, the loop returns to the generator, but what goes back is not the verdict, it is the specific unmet criteria. A generator told only “rejected” has nothing to change, so it resamples instead of revising. The dotted max rounds edge is the one people forget to build: when the round cap is reached, the loop exits anyway, carrying the best draft and a passed=False flag. Both pass and max rounds reach Output; a system that cannot tell you which one it took is worse than no evaluator at all.

Use when: you have clear, articulable criteria and iteration measurably helps, literary translation judged against a style guide, code that must pass a linter, or a research answer that has to cover N named sources.

Failure mode: oscillation. With vague criteria, round 3 undoes round 2 while the bill keeps rising:

r1: draft mentions no caveats
r2: judge says "add caveats"        → draft adds three paragraphs of caveats
r3: judge says "too hedged"          → draft removes all caveats
r4: judge says "add caveats"         → ...

Three fixes:

1. Cap the rounds.
2. Make the evaluator return structured per-criterion verdicts, not prose, so “too hedged” cannot silently replace “add caveats”.
3. If a criterion can be checked in code, check it in code. Whether the tests pass, whether the JSON parses, and whether a required section exists are all questions a program answers exactly. An LLM judge is the fallback for genuinely subjective dimensions, not the default.

Cost. 2N× for N rounds, strictly 2N+1 calls, since the first draft is generated before the loop begins. Empirically rounds 1–2 capture most of the gain; beyond 3 you are usually paying for churn.

There is a standing cost lever here too: cache the judge’s system prompt. Prompt caching means the provider stores its precomputed internal state for the leading, unchanged part of your prompt and bills a repeat of that part at roughly a tenth of the normal input rate (Prompt caching derived). The judge’s system prompt is byte-identical every round, so rounds 2 and later read the criteria at ~0.1× input cost. One caveat: the cached prefix has to clear the model’s minimum (512 tokens on claude-opus-5), or nothing caches, with no error, only cache_creation_input_tokens: 0 in the usage block. A terse judge prompt plus three bullets will not reach it; the full criteria list that makes the pattern work usually will.

This pattern is coded end to end in Building One: Evaluator–Optimizer, Three Tiers, whiteboard pseudocode, a LangGraph version where the retry is a graph edge, not a while, and a production version against the Anthropic SDK with the judge’s prompt cached. It also shows which criteria to move out of the judge and into ordinary code.

ReAct — Reason + Act

The model reasons about what it needs, calls a tool, reads the result, and repeats until it decides it is finished. That loop is what most people mean by “agent”.

The shape, in plain words. ReAct interleaves three things inside one growing conversation:

• a thought about what is needed next,
• an action that calls a tool,
• an observation, which is that tool’s result fed back in.

Then it thinks again with the result in hand, and repeats.

The problem it solves. Some tasks cannot be planned in advance, because what you do third depends on what the second step returned. ReAct refuses to decide anything before it has to.

flowchart TD
    S([Goal]) --> T[Thought<br/>what do I need?]
    T --> A[Action<br/>tool call]
    A --> OB[Observation<br/>tool result]
    OB --> D{Enough?}
    D -->|no| T
    D -->|yes| F([Answer])

    style S fill:#1d3557,color:#fff
    style F fill:#2d6a4f,color:#fff

The division of labour matters: the model produces the thought and the action, and your code runs the tool and returns the observation. Every lap appends to the same conversation, and that one fact is what the cost derivation below turns on.

Mechanism, modern form. You do not hand-prompt Thought:/Action:/Observation: any more, because native tool calling is ReAct. The three parts map onto the API like this:

• the thought is the model’s reasoning, emitted as ordinary text;
• the action is a tool_use block, a structured request to run one of your tools;
• the observation is a tool_result block you send back.

The tool_use block is emitted under constrained decoding, meaning the runtime blocks any token that would break the tool’s declared shape, so the call always parses (Structured output is a guarantee, not a request).

The pattern is named after ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022, arXiv:2210.03629). What the paper proposed as a prompt format (the model made to literally emit the strings Thought:, Action: and Observation:) the API now provides as a message structure. The idea survived; the string formatting did not.

That mapping is easiest to see in a real message array. A message array is the list of alternating turns you resend on every request. This one is a single turn of debugging “the checkout endpoint 500s on empty carts”. The diagram’s first two boxes arrive as one assistant message holding two content blocks:

resp.content == [
    TextBlock(text="The 500 is probably in the totals path. Let me read the handler."),   # Thought
    ToolUseBlock(id="toolu_01A", name="read_file",                                        # Action
                 input={"path": "api/checkout.py", "lines": "40-80"}),
]

The observation is what you append on the next request, and it is a user-role message. Roles are how the API labels who produced each turn: assistant for the model’s output, user for everything you send in. The environment has no role of its own, so tool results go under the user role. The assistant message is appended back verbatim, both blocks unedited, and tool_use_id ties the result to the request that asked for it:

messages.append({"role": "assistant", "content": resp.content})      # BOTH blocks, verbatim
messages.append({"role": "user", "content": [
    {"type": "tool_result", "tool_use_id": "toolu_01A",              # Observation
     "content": "43: total = sum(i.price for i in cart.items)\n44: return total / len(cart.items)"},
]})

And the Enough? diamond is not a node you write. It is resp.stop_reason on the next response: "tool_use" is the no edge back to Thought, and "end_turn" is the yes edge to Answer. That is the entire loop. There is no ReAct library; there is a while on a stop reason. A worked trace walks the same array end to end, and the code lab builds the loop as a runnable exercise.

Use when: the number and order of steps genuinely depend on what the environment returns, debugging, search, and anything else where step 3 is unknowable until step 2 has run.

Failure mode 1 — the loop

The agent calls the same tool with the same arguments three turns running. The mechanism explains the fix: the model’s context now contains three near-identical (call, result) pairs, and a model predicting the next token from a context full of a repeated pattern makes the fourth repetition more likely, not less. The loop reinforces itself.

The key move is to feed a trip message back before halting (detection and the break are in Reliability and guardrails). A trip message is text your loop-detector writes into the conversation the moment it notices the repetition, not an error and not a halt, but a plain observation:

you have called search_docs with the query rate limit three times and received the same result each time; that approach is not working, try something else.

You append it exactly where a tool result would have gone and let the loop take one more lap. The repetition is self-reinforcing only because the context contains a pattern and nothing contradicting it, so inserting the contradiction is the cheapest thing that can break it. Halt only if the next lap repeats anyway.

Failure mode 2 — context blowup

Every observation is appended forever, so turn n sends P + a·n input tokens. Per-turn cost therefore grows linearly in n; it is the running total that is quadratic, at n·P + a·n²/2. Calling the per-turn curve quadratic is the usual slip, and it matters because it changes the fix: trimming old observations attacks a and shrinks both terms, but the only lever on n² is capping the steps.

With a 2k-token base prompt and 1.5k per step, turn 2 sends about 5k tokens and turn 20 about 32k, a 6.4× growth, not the often-quoted 10× (that figure is the limit with the base prompt dropped entirely). Either way, the second half of this failure is worse than the bill: at turn 20 the instructions sit in the middle of a 32k window, the worst position on the recall curve (Why quality degrades in long contexts). That is why long ReAct sessions drift.

Cost. 5–20×, unbounded without a cap. This is the pattern that needs guards most, and the only one where “add a step cap” is non-negotiable, not good hygiene.

Plan-and-Execute

Separate deciding from doing and the bill changes shape: one expensive call writes the entire plan up front, cheap calls carry out the steps one by one, and the plan is rewritten only when a step fails. The gap between this and ReAct is not a fixed multiple. It widens with the length of the task.

The shape, in plain words. Look around briefly, plan the whole thing once with a strong model, then execute the steps with a cheaper one, and replan only on failure.

The problem it solves. ReAct pays for its strongest model to make every decision, inside a conversation that keeps growing, when in fact most steps are simple instruction-following. Deciding is hard and happens once. Following a written instruction is easy and happens n times.

flowchart TD
    G([Goal]) --> RECON[Recon: 1-2 cheap<br/>reads of the environment]
    RECON --> P[Planner<br/>Opus, once]
    P --> Q[[Step queue]]
    Q --> E[Executor<br/>Haiku/Sonnet, per step]
    E --> C{Step OK?}
    C -->|yes, more left| Q
    C -->|yes, done| F([Result])
    C -->|no| RP{Replans < 3?}
    RP -->|yes| P
    RP -->|no| X([Fail loudly])

    style G fill:#1d3557,color:#fff
    style RECON fill:#40916c,color:#fff
    style P fill:#40916c,color:#fff
    style F fill:#2d6a4f,color:#fff
    style X fill:#9d0208,color:#fff

Three boxes carry the whole cost argument. Recon is a couple of cheap look-first calls, listing the directory, reading the schema (the declared shape of the data: its tables and columns), whose only job is to make sure the plan is written against reality, not a guess. Planner Opus, once says the strong model is invoked exactly one time per plan, not once per step; that is where the intelligence budget goes. Step queue is a real, inspectable data structure, a list you can print, diff, and hand to a human before a single side effect happens. The executor is the cheap tier, and it is cheap because the queue already made the decisions.

Step OK? fans three ways. yes, more left pops the next item and keeps the same plan; yes, done exits with a result; no goes to the replan counter. Replans < 3? is a hard stop whose no edge goes to fail loudly: a terminal that reports the partial state and the last error instead of returning something plausible. An agent that quietly returns a half-done result on plan exhaustion is indistinguishable from one that succeeded, the worst property a long-running system can have.

Mechanism: why it is cheaper than ReAct over the same number of steps. Two reasons. First, each executor step runs on a short context, the plan plus this one step, so nothing accumulates and there is no quadratic growth. Second, execution runs on a smaller model, because following an explicit instruction is much easier than deciding what to do next.

Use when: the horizon is long, many steps separate start from finish, the environment is stable enough that a plan written now is still true in ten steps, and a human may want to read the plan before anything executes. That last one is often the real reason.

Failure mode: the stale plan. The plan was written before the environment was understood, so step 4 assumes a file that does not exist. The recon phase exists precisely for this: one or two cheap reads before planning. And cap replans at 3, a third failed plan means the goal is wrong, not the plan.

Cost: the gap is not a fixed multiple. Both patterns take the same P, a, and n; only the accumulation differs. ReAct grows as n·P + a·n²/2: quadratic in n. Plan-and-Execute collapses to roughly 9,000 + 2,300·n on the same inputs, linear in n (a couple of recon reads, one planner call, then n short executor calls). Because one term is quadratic and the other linear, the ratio grows with the horizon:

n =  5   ReAct  28,750   P&E  20,500   →  1.4×
n = 12   ReAct 132,000   P&E  36,600   →  3.6×
n = 25   ReAct 518,750   P&E  66,500   →  7.8×

At five steps the patterns are within 40% of each other and the extra machinery is not worth it. At twenty-five, ReAct is reading half a million tokens to do the same work. (The summary table at the end reports 5–20× for ReAct and 3–8× for Plan-and-Execute, each against a single plain call, a different denominator from the ratios above, which compare the two patterns to each other.)

Model tiering multiplies the gap. At n = 12, only the ~9,000 recon-plus-planner tokens need the strong model; the ~27,600 executor tokens run on Sonnet. Priced at input rates, all-Opus ReAct is about $0.66 against Plan-and-Execute’s $0.13, so 5.2× cheaper in dollars against 3.6× in tokens, because the cheap tier absorbs the bulk.

The table below sets ReAct and Plan-and-Execute against the five axes that decide between them. The one Plan-and-Execute wins outright is human can review before execution, and it is often the reason the design is chosen at all.

The hybrid, planning coarsely and running ReAct within each step, is what most production coding agents actually do.

Reflexion

When an attempt fails, a separate call writes down why in one sentence, that sentence is stored, and every later attempt starts with it in hand. Two versions of that idea share the diagram below, and they have very different bills.

The shape, in plain words. Reflexion is the draft-and-critique loop of Evaluator–optimizer plus memory that survives the attempt: the lesson outlives the retry that produced it.

The problem it solves. A plain retry loop re-learns the same fact every time. It discovers on attempt 2 exactly what it discovered on attempt 1 of the previous run, and pays for the discovery twice. Writing the fact down converts a failed attempt from wasted spend into something reusable.

The diagram is a retry loop with one extra box, the Lesson store cylinder. Watch which edge writes to it and which reads from it.

flowchart TD
    T([Task]) --> A[Attempt]
    A --> EV{Succeeded?}
    EV -->|yes| O([Done])
    EV -->|no| R[Reflect:<br/>why did it fail?]
    R --> M[(Lesson store)]
    M --> A

    style T fill:#1d3557,color:#fff
    style O fill:#2d6a4f,color:#fff
    style M fill:#7209b7,color:#fff

Mechanism. Attempt runs the task. Succeeded? is a check on the outcome of acting, not on the thing produced: the tests passed, the query returned the right row. On yes the loop exits without writing anything. Only the no edge reaches Reflect, a separate call whose job is not to fix the task but to name the cause in one falsifiable sentence. That sentence, and nothing else, is written to the store. The arrow back to Attempt is retrieval: the next attempt’s prompt is assembled as task plus retrieved lessons, with the lessons injected into the system prompt ahead of the task, so they read as standing constraints.

Three mechanics that arrow hides:

• When the write fires. Only on failure, and only after reflection, never on success. Otherwise the store fills with descriptions of things that already work.
• What gets retrieved. Usually all of them, under a hard cap of a few dozen short lines. Searching such a small store by meaning mostly returns noise and can silently drop the one lesson that mattered. Graduate to a search only once the cap starts biting.
• Where they land. In the system prompt, before the task, in a fixed order, otherwise the leading text changes every run and you lose prompt caching on the one block that never changes.

How Reflexion differs from Evaluator–Optimizer is not really “memory”. Evaluator–Optimizer critiques the thing produced against written criteria, while Reflexion reflects on the outcome of acting in an environment, so it needs a real attempt to have happened and failed. That is why its success check is a test run, not a rubric.

Two scopes — within-task and cross-task

The name is ambiguous, and the cost line depends on which one you mean.

Within-task Reflexion: the store lives for one task, holds a handful of lessons about this task’s environment, and is thrown away at the end. Cost is bounded by the retry cap, and nothing persists.

Cross-task Reflexion: the store outlives the task and is loaded by every future run. This is the version that pays for itself, and the version that charges rent forever. The cost line below is this one.

Cross-task only pays off if lessons transfer: a lesson learned on task A is still true and relevant on task B. That holds when the tasks come from the same distribution: one codebase, one schema, one API, one toolchain. “Fix bugs in this repo” transfers, because the table names do not change between tickets, and neither does the CI matrix (the set of language versions the project’s automated build tests against). “Answer arbitrary user questions” does not transfer, and a store built from it is pure overhead.

Failure mode: junk lessons. The store fills with unfalsifiable advice:

✗ "Be more careful with edge cases."          unfalsifiable, pure token tax
✗ "Think step by step."                        already true, adds nothing
✓ "orders_v2 is canonical; orders is a stale view kept for BI."
✓ "The CI matrix runs Python 3.10 — no match statements in runtime code."

The good ones are specific, non-obvious from the code, and falsifiable: you could run a check that proves the lesson wrong, which is exactly what “be more careful” can never be. (BI is business intelligence, the reporting stack.)

Here is one full cycle producing that third line, so the arrows are not abstract:

attempt 1   SELECT sum(total) FROM orders WHERE day = ...   → $41,220
            harness check: expected $52,900 from finance     → FAIL
reflect     "the query ran and returned a number, so this is not a syntax
             error; `orders` is missing rows that `orders_v2` has"
store  +=   "orders_v2 is canonical; orders is a stale view kept for BI."
attempt 2   system prompt = [3 prior lessons + the new one] + task
            SELECT sum(total) FROM orders_v2 WHERE day = ... → $52,900  ✓

Note what did not happen: attempt 2 was not handed the failing query to patch. It was handed the same task plus a fact, and re-derived the query from scratch. That is why the lesson also fixes every later query against that table, where patching the SQL would have fixed exactly one. Enforce specificity in the reflection prompt, cap the store, and let it evict.

Cost. 3–10× for the retry loop, plus a standing per-run cost that is easy to underestimate. A 40-lesson store of ~25 tokens each is a 1,000-token block; resent on all 20 turns of a run, that is 20,000 input tokens, or about $0.10 per run on Opus, $0.01 if the block sits in a cached prefix. At 10,000 runs a day that is the difference between $1,000/day and $100/day, for text nobody reads. That $900/day gap is the entire argument for keeping the store in a stable, cacheable position, capping its size, and deleting lessons that stop earning.

Autonomous Loop

One pattern in this chapter has no human inside it: the agent writes its own to-do list, works through it, and rewrites it as it learns. That arrangement has a failure all its own, drift, where every individual step is defensible and the chain of them is not, and three conditions that make it survivable at all.

The shape, in plain words. Give the loop a goal and a budget. It generates its own list of tasks, executes them one at a time, checks each result, and writes what it learned back into the list (appending, reordering and deleting tasks) until the goal is met, the budget runs out, or a drift check halts it.

The problem it solves. A goal too open-ended to enumerate up front, where the second task genuinely cannot be written until the first one has reported. The running example is “reduce our p95 latency”, p95 latency being the response time that 95% of requests come in under, the standard way to talk about the slow tail, not the average. The cost of solving it this way is that nothing in the loop’s own logic makes it stop.

The diagram has three exits, Done, Halt + partial report, Halt + alert, and two diamonds in the middle, Machine-checkable verification and Drift vs ORIGINAL goal, that make the rest survivable.

flowchart TD
    G([Goal + budget]) --> P[Generate tasks]
    P --> Q[[Task queue]]
    Q --> EX[Execute next]
    EX --> VER{Machine-checkable<br/>verification}
    VER --> CR[Self-critique]
    CR --> U[Update queue]
    U --> DR{Drift vs<br/>ORIGINAL goal}
    DR -->|drifted| D([Halt + alert])
    DR -->|ok| ST{Stop?}
    ST -->|goal met| F([Done])
    ST -->|budget out| B([Halt + partial report])
    ST -->|no| Q

    style G fill:#1d3557,color:#fff
    style F fill:#2d6a4f,color:#fff
    style B fill:#bc6c25,color:#fff
    style D fill:#9d0208,color:#fff

Mechanism. The budget is an input, not a setting, because it is the only thing that makes termination guaranteed. Generate tasks takes the goal, and, on later laps, what the last execution revealed, and emits the initial queue. Execute next pops one item and runs it, usually as a small ReAct or Plan-and-Execute sub-run, which is where the cost multiplier comes from.

Then two checks, and the reason there are two is worth stating. Machine-checkable verification asks did this work, using something that needs no LLM: p95 measured before and after, the test suite exit code, the row count. It is ground truth, cheap and incorruptible, but it can only score the task that was actually run. Self-critique asks the question no assertion can: was that the right task at all, and what does the result imply about the next one? It is judgement, a model call, and fallible. You need both, because verification can pass on a task that should never have been queued, the dashboard got built, the tests are green, and the p95 is unchanged.

Update queue then writes the consequence back: append what execution revealed, reprioritise what the result made urgent, drop items the result made pointless. That single change, a queue that grows from its own output, is what makes drift possible, and it is why this pattern needs a goal-comparison check that Plan-and-Execute (whose queue is fixed at plan time) does not.

Failure mode: drift. Each self-generated task looks locally reasonable; twenty steps later the agent is optimizing something nobody asked for. Watch the subject change from latency to dashboards, one small step at a time:

goal: reduce p95 latency
  → profile endpoints        ✓
  → add caching              ✓
  → refactor the cache layer  ~
  → add cache metrics         ~
  → build a metrics dashboard ✗
  → add dashboard auth        ✗✗

Every arrow is defensible on its own. The chain is not. Detection: the Drift vs ORIGINAL goal diamond periodically re-scores the queue against the immutable original goal text, never the latest restatement, because restating is exactly how drift launders itself. Ask a cheap model, per queued task: “On a 0–1 scale, how directly does completing this task reduce p95 latency? Quote the clause of the goal it serves.” Scored against the frozen string, the six tasks come out like this:

profile endpoints          1.0   "reduce p95" — measures the target directly
add caching                0.9   plausible causal path to p95
refactor the cache layer   0.5   maintains a thing that helps; no p95 delta
add cache metrics          0.4   observes the helper, doesn't move p95
build a metrics dashboard  0.1   ← first task that cannot quote the goal
add dashboard auth         0.0   serves the dashboard, not the goal

The first task that cannot quote a clause of the goal is the breach: here build a metrics dashboard, step 5, not step 6 where the wrongness becomes obvious to a human. Trip at a threshold and take the drifted edge to Halt + alert: halt if the top queued item scores below 0.3, or if the queue’s mean score drops for two consecutive laps. Note what would have happened had you re-scored against the agent’s current framing, by now something like “improve latency observability”, every one of these six tasks scores above 0.8 against that. That is the laundering, and freezing the goal string is the only defence.

Three halts, not interchangeable. Done on goal met is success. Halt + partial report on budget out is a normal exit the loop is designed to hit, so the report has to be genuinely usable: what was completed, what is half-done, what the queue still holds. Halt + alert on drifted is the only one that pages a human, because it means the loop’s own judgement is no longer trustworthy and no further self-critique will catch that.

Cost. 20–100×+, and unbounded by construction: Update queue can always add one more item. A hard dollar ceiling is not a safety feature bolted on; it is part of the architecture, and it is what turns Halt + partial report from an error path into a designed outcome. Make it concrete: at $0.15 per sub-run and a $30 ceiling, that is 200 laps. If the goal plausibly needs more, the budget is wrong, and you find that out by reading a partial report.

Fully autonomous is usually the wrong choice. The three conditions that make it survivable are reversible actions, a machine-checkable success signal, and a hard budget; without all three, do not delete the human. The autonomous-agent case study is the full build.

Side by side

All nine patterns fit in one table. Two of its columns mislead if read naively: “cost” is a token multiplier, not a dollar figure, and “debuggable” is a claim about replay, not about logging.

Cost is total input plus output tokens at a fixed model tier, relative to answering the same task in one plain call, not dollars. Tiering is a separate lever that moves dollars underneath a fixed multiplier, which is why Routing reads above 1× here (its 205-token classification riding on a handler call of 2,400 tokens gives (205 + 2,400)/2,400 ≈ 1.09×) and below 1× in dollars (0.53×), once the label has moved traffic onto a cheaper rung.

Debuggable counts how many model-driven decisions sit between input and output, and whether you can replay a failure deterministically. Five stars means every branch was taken by your code, so a failing case reproduces exactly. One star means the model chose the path, the tasks, and when to stop, so the same input may not even fail the same way twice.

The context-growth column is the one to read first. Only ReAct is quadratic, and it is also the default choice most people reach for, and that mismatch is where most agent cost surprises come from.

Cost and Debuggable are worth reading together at the ends. The cheapest rows are the most debuggable (Prompt Chaining and Routing, five stars); the most expensive is the least (Autonomous, 20–100× and one star). That is not a coincidence: every star you lose is a decision you handed to the model, and every decision handed to the model is a call you now pay for and cannot replay.

The middle is not monotonic, and the exceptions are informative. At the round cap this chapter recommends (N ≤ 3, so 2N ≤ 6×), Evaluator–Optimizer is cheaper and more debuggable than Orchestrator–Worker. And Plan-and-Execute costs less and debugs better than ReAct, because it makes all of its decisions in one place, a single planner call producing an inspectable queue, not one decision per turn scattered through a growing conversation. The lesson from both: debuggability tracks where the model decides, not only how much you pay.

Real systems compose

No production system is one pattern. The support agent below wires four of them together, a router in front, two different lanes behind it, one shared quality gate, and a single exit to a human. Trace any path from Message to an exit and you cross a router, then one handler, then a gate.

flowchart LR
    M([Message]) --> R{Router}
    R -->|FAQ| RAG[Single RAG call]
    R -->|account| RE[ReAct + tools]
    R -->|angry| H([Human])
    RE --> EV{Grounding gate}
    RAG --> EV
    EV -->|pass| O([Reply])
    EV -->|fail| H

    style R fill:#bc6c25,color:#fff
    style EV fill:#2d6a4f,color:#fff
    style O fill:#2d6a4f,color:#fff

Read it as a chain of four patterns: a router in front, then either a single retrieval-augmented (RAG) call or a ReAct loop, then an evaluator acting as a gate, and finally an escalation to a human. Every box was chosen against a specific alternative.

Why each branch is what it is

The router’s three classes buy three different cost and risk profiles, not three prompts.

FAQ (a frequently-asked question, the kind a published policy already answers) goes to a single RAG call: one retrieval, one generation, no loop. A policy question has a document that answers it, and an agent loop would only add ways to be wrong.

The account class goes to ReAct with tools, because “why was I charged twice” cannot be answered without looking up this user’s actual charges, and step 3 depends on what step 2 returns.

The angry class exits to a human immediately. No model composes a reply, so the branch where a bad answer would do the most damage is also the branch with the least automation.

The grounding gate, and its fail edge

The grounding gate is the Evaluator–Optimizer of Evaluator–optimizer reduced to a single round: it judges once and never asks for a revision. It checks exactly one thing: is every factual claim in the draft reply supported by a document that was actually retrieved, or by a tool result actually returned in this session? Not “is this a good answer”, which is the vague criterion the earlier section warns about. Every claim must cite a span from the retrieved chunk or a field from the tool response, and an uncited claim fails. That makes the check mechanical on the tool branch, no LLM judge needed, and near-mechanical on the RAG branch.

Both lanes pass through it, because each hallucinates in its own way: the RAG lane embellishes beyond the passage it retrieved, and the ReAct lane narrates a tool result it never actually got. One gate catches both.

The fail edge is the part most designs omit. A grounding failure does not retry, and it does not ship with a hedge. It goes to the human queue, the same terminal as the abusive messages. That is the whole safety argument in one arrow: the system’s answer to “I cannot support this claim” is a person, not another sample. It also tells you what to monitor: the escalation rate by branch. Rising on the FAQ lane means retrieval is missing documents; rising on the account lane means a tool is failing quietly.

Conclusion

• The choice is about your task, not the patterns. Distinct input categories → route. A decomposition you can name at design time → chain, parallelize, or (if the count varies) orchestrator–worker. Scorable output → evaluator–optimizer. Otherwise a true agent, ReAct or Plan-and-Execute, split by how often it must observe.
• Start with the simplest tier and escalate only on measured failure. Every decision you hand the model costs tokens you cannot replay, which is why the cheapest patterns are also the most debuggable.
• Cost is a token multiplier at a fixed tier; tiering is a separate lever on dollars. The two compose, routing plus tiering plus caching is where the large savings live.
• The dangerous term is quadratic context growth, and only ReAct has it. A step cap is non-negotiable there; Plan-and-Execute avoids it by keeping each step on a short context.
• Reflexion and the Autonomous Loop are add-ons, not base choices. Reflexion adds transferable memory; the Autonomous Loop deletes the human and needs reversible actions, a machine-checkable success signal, and a hard budget before it is safe.
• Real systems compose several patterns, and a grounding gate whose failure edge goes to a human is what keeps the composition honest.

Further reading

• Anthropic, Building Effective Agents, the source of the workflow-vs-agent framing and of prompt chaining, routing, parallelization, orchestrator–workers, and evaluator–optimizer.
• Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, 2022, arXiv:2210.03629.
• Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning, 2023, arXiv:2303.11366.

Next: Tools & MCP. Also from this chapter: Building One: Evaluator–Optimizer, Three Tiers writes the Evaluator–optimizer loop three times over, whiteboard pseudocode, LangGraph, and the Anthropic SDK.