02 — Design Patterns

There are nine standard ways to wire large language model (LLM) calls together into a working system — from a fixed pipeline whose every step you wrote by hand, to a loop that invents its own next task.

Each of the nine gets the same four things: what the shape is, what problem it solves, what breaks, and what it costs.

Read it end to end and “design an agent that does X” stops being a blank page. You will be able to name the pattern, name the alternative you rejected, and defend both with arithmetic rather than taste.

Who this is written for. The immediate audience is someone preparing for a system-design interview round that opens with that exact prompt, so several passages below say what to claim out loud and what a good interviewer will probe next.

Nothing in it is interview-only. The patterns, the failure modes and the cost derivations are the same ones you need to actually ship the system. The reason the interview asks for them is that they are what the job is.

What all nine have in common

Before any pattern, fix the shape of the whole thing: what goes in, and what comes out.

All nine take the same input — a task written in ordinary text, such as “refund order 4021” or “find out why checkout returns a 500 error on empty carts”.

All nine produce the same output — one final answer in text, plus whatever side effects their tools performed on the way: a refund issued, a file edited.

A tool is a function you wrote and then described to the model — issue_refund(order_id), read_file(path), search(query). The model cannot run it. It emits a structured request to call one, your code executes that request, and you feed the result back to the model as more text. Tools are the only way a model touches anything outside its own output, and the word appears on nearly every page below.

So nothing about the input or the output tells the nine apart. What differs is entirely the wiring in between: how many model calls happen, who decides how many, and how much of the accumulated conversation each call is allowed to see.

A design pattern here is a control-flow diagram over model calls, and that is all it is.

Four terms the rest of the chapter leans on

Pin these down before the first pattern, because every cost argument below is written in them.

Token. The unit a model reads and bills in — roughly four characters of English. Run that conversion once so the later numbers are checkable. An English word averages about 5.2 characters once you count the space after it, so a 500-word page is 500 × 5.2 = 2,600 characters, and 2,600 / 4 ≈ 650 tokens.

Context. Everything sent along with one call: the system prompt (the standing instructions that set the model’s role and rules), the tool definitions, the conversation so far, and the new request.

Context window. The hard cap on how large that context may get — the maximum number of tokens a single call can hold, prompt and answer together. Exceeding it is an error, not a gradual degradation.

Stateless. The interface you call remembers nothing between calls, so a multi-step agent has to resend the entire conversation on every step. That last fact is why context size, and therefore cost, is the axis every pattern in this chapter is really trading against.

How to read the N× cost numbers

Every N× below is unreadable without two symbols, and both are used everywhere from here on.

• P — the fixed prefix. The system prompt plus the tool definitions. Paid in full on every single call, and it never grows.
• a — the per-turn delta. What one lap of a loop adds to the conversation: the model’s own reply plus the tool result it triggered.

With those, 1× is one plain LLM call on a prompt of size P. And N× means the pattern reads and writes N times as many tokens as that one call would have, answering the same task.

N× is a token multiplier at a fixed model tier, not a dollar figure. Moving work onto a cheaper model changes the dollars underneath a fixed multiplier.

That is exactly why Routing reads ~1.09× in the summary table at the end of the chapter — the classifier’s ~205 tokens riding on top of a handler call that was going to happen anyway — and 0.53× in dollars, once the label it produced has moved most of the traffic onto a cheaper rung (Routing).

The growth formula, read one term at a time

Derivations assume the chapter 00 model. Because the interface is stateless, history is resent every turn, so an n-turn loop sends this many input tokens over the whole run:

n·P + a·n²/2

n·P is the prefix term. Each of the n turns resends the fixed prefix P, so you pay P exactly n times.

a·n²/2 is the history term. Turn t additionally carries the a tokens every earlier turn contributed, so the a-terms accumulate as a·(1 + 2 + 3 + ⋯ + n). That arithmetic series is worth exactly n(n+1)/2, which this chapter rounds down to n²/2 to keep the shape visible.

That rounding is not a half-turn’s worth; it is a·n/2. Substitute a = 500 and n = 20:

exact history term    500 · 20·21/2 = 105,000 tokens
rounded history term  500 · 400/2   = 100,000 tokens
discarded              5,000 tokens

5,000 is about 5% of the 105,000 a-terms. Take a prefix of P = 2,000 and the run’s exact total is 20 × 2,000 + 105,000 = 145,000 tokens, so the 5,000 is about 3% of that. The approximation is fine for arguing about growth and wrong for quoting a bill.

The second term grows with the square of the turn count — double the turns and it roughly quadruples. Whenever this chapter says “quadratic growth” it means exactly that term.

One bookkeeping warning, because neighbouring chapters round differently. Chapter 00’s convention footnote records the two ways of counting the first turn’s delta; both round to a·n²/2, and the rule is never to mix them inside one calculation.

For the same reason, note that Why the cost gap is so large carries the exact n(n+1)/2 where this chapter carries the n²/2 approximation. On the same inputs — P = 2,000, a = 500, n = 20 — chapter 01 reports 145,000 tokens and this chapter’s formula gives 140,000. Same shape, two levels of rounding. Do not mix a figure from one into a calculation from the other.

How each pattern below is laid out

Every pattern is presented in the same order: the shape in plain words, the diagram and how to read it, the mechanism, when to reach for it, the failure mode with a concrete trace — and only then the cost.

Where the cost is a clean function of the wiring, it is derived in front of you and you should check the arithmetic. Routing’s 0.53×, Orchestrator–Worker’s 4–15×, and the ReAct-versus-Plan-and-Execute gap are all built from named quantities.

Where a pattern’s multiplier depends mostly on how long you let it run — ReAct, Reflexion, the Autonomous Loop — the headline figure is a range observed in the field, and the section says which lever moves you within it rather than pretending the range fell out of algebra.

Picking one

You can choose among the nine before you know any of them in detail, because the choice turns on properties of your task, not of the patterns. The decision tree below asks the questions that actually discriminate, glosses each pattern in plain language as it reaches it, and labels how much freedom each choice hands to the model.

Read the tree top to bottom. Each diamond is a question about your task; each rectangle is a pattern you land on. The second line inside every rectangle names how much freedom that pattern hands the model — pure workflow, bounded agency, or true agent. Do not worry that the pattern names mean nothing yet; the paragraphs after the diagram gloss each one as the tree reaches it.

flowchart TD
    S0{Distinct input categories<br/>wanting different handling?} -->|Yes| ROUTE["Router<br/>pure workflow"]
    ROUTE -.->|then pick a pattern<br/>per branch| Q
    S0 -->|No| Q{Is the task<br/>decomposable up front?}
    Q -->|Yes, fixed steps| CHAIN["Prompt Chaining<br/>pure workflow"]
    Q -->|Yes, independent steps| PAR["Parallelization<br/>pure workflow"]
    Q -->|Yes, but count varies| ORCH["Orchestrator–Worker<br/>bounded agency"]
    Q -->|No| DYN{Is output quality<br/>cheaply measurable?}
    DYN -->|Yes| EVAL["Evaluator–Optimizer<br/>bounded agency"]
    DYN -->|No| ENV{Must it observe<br/>before every decision?}
    ENV -->|Yes| REACT["ReAct<br/>true agent"]
    ENV -->|No, checkpoints suffice| PLAN["Plan-and-Execute<br/>true agent"]

    style CHAIN fill:#2d6a4f,color:#fff
    style PAR fill:#2d6a4f,color:#fff
    style ROUTE fill:#2d6a4f,color:#fff
    style ORCH fill:#40916c,color:#fff
    style EVAL fill:#40916c,color:#fff
    style REACT fill:#95d5b2,color:#000
    style PLAN fill:#95d5b2,color:#000

Question 1 — distinct input categories

The first question is whether you have distinct input categories wanting different handling. A refund request and a stack trace want different prompts, different tools, and plausibly different models.

If yes, put a router in front: one cheap classification call that reads the input, labels it, and hands it to the branch built for that label (Routing).

Routing is not a rival to the other eight. That is what the dotted edge back into the tree means: then pick a pattern per branch. A router feeding a RAG lane and a ReAct lane is one design, and saying so is most of what separates a senior answer from a junior one. (RAG is retrieval-augmented generation: fetch the relevant documents first, then answer only from them — chapter 05.)

Question 2 — decomposable up front

If you have no distinct categories, the tree’s second question is whether the task is decomposable up front: can you say now what the pieces are? There are three flavours of yes, and the distinction between them is the whole left half of the tree.

Yes, fixed steps — you can name every step at design time — is Prompt Chaining, a fixed pipeline in which each call’s output becomes the next call’s input (Prompt chaining).

Yes, independent steps — the steps exist but none reads another’s output — is Parallelization, where you fan the work out across calls that run at the same time and merge their results (Parallelization).

Yes, but count varies is Orchestrator–Worker (Orchestratorworker). You know the shape of the decomposition, but not how many pieces this particular input needs. So one manager call — the orchestrator — splits the job into however many pieces the input demands, and hands each piece to a worker that starts from a blank context. That varying count is precisely why a model, and not your code, has to do the splitting.

Question 3 — cheaply measurable output quality

If the task is not decomposable, ask whether you can score the answer with something cheap: a linter (a program that flags style and correctness problems in code), a test suite, a checklist of required sources.

If you can, reach for Evaluator–Optimizer (Evaluatoroptimizer) — a draft-and-critique loop in which one call writes, a second call judges the draft against written criteria, and the writer revises until it passes or the round cap is hit. It buys you iteration you can actually trust, because something other than a vibe decides when to stop.

Question 4 — observation frequency

If quality is not measurable either, the last question is not whether the pattern observes the environment. Both remaining options do. The question is how often.

ReAct, short for Reason + Act, re-reads the world between every step, because step 3 is unknowable until step 2 returns (React reason act).

Plan-and-Execute — write the whole plan first with a strong model, then run the steps with a cheap one (Plan and execute) — takes the No, checkpoints suffice branch. It observes too: its own diagram opens with a reconnaissance phase and gates every step on a result. But it observes only at fixed points, once before planning and once per step boundary.

The discriminator is observation frequency, not observation at all. Picking wrong here is the single most common mis-selection in this chapter, because “it needs to look at things” is true of both.

Three tiers of freedom

The tree shades the boxes in three greens, and the second line of every box names its tier in words. Here they are in order of how much the model gets to decide.

Pure workflow — Prompt Chaining, Parallelization, Router, the dark green boxes. You wrote the control flow, and the model only fills in the individual steps.

Bounded agency — Orchestrator–Worker, Evaluator–Optimizer, mid green. The model gets one specific decision: how many workers to spawn, or whether to iterate again. The shape is still fixed by you — decompose, fan out, synthesize; generate, judge, revise — and so is the ceiling on how far it can go.

True agent — ReAct, Plan-and-Execute, pale green. The model chooses what to do next, and nothing bounds that except a guard you remembered to add.

Start with a pure workflow. Escalate to the next tier up only when the simpler one demonstrably fails — and be able to say what “fails” means: a measured accuracy floor on a named input slice, not a vibe.

Why two patterns are missing from the tree

Reflexion and Autonomous Loop are absent on purpose. The tree picks a base loop, and neither of those is one.

Reflexion (Reflexion) — a reflection loop that writes down why an attempt failed and carries that note into the next attempt — is a modifier. It is Evaluator–Optimizer (or ReAct, or Orchestrator–Worker) plus a store of lessons that survives the attempt. So you choose a base first, then decide separately whether those lessons transfer.

Autonomous Loop (Autonomous loop) is what a pale-green box becomes when you delete the human and let the loop generate its own next task. That is a governance decision — reversibility, budget, a success signal a program can check without a model — not a decomposition one.

Putting either in the flowchart would hide the question that actually decides it.

1. Prompt Chaining

The simplest pattern in the chapter is a fixed pipeline of model calls, and one check keeps it honest: a gate that compares each step against the original input rather than against the step before it.

The shape, in plain words. A prompt chain is a fixed sequence of calls in which each call’s output becomes the next call’s input, with ordinary code between the steps deciding whether the run continues. You wrote the sequence at design time; the model only fills in the individual steps.

The problem it solves. One call asked to do four things at once does all four adequately and none well. Four calls each asked to do one thing are each easy to prompt, easy to test, and easy to price.

The diagram below shows a three-step chain. The thing to look at is the branch out of the Gate.

flowchart LR
    I([Input]) --> S1[Step 1] --> G{Gate} --> S2[Step 2] --> S3[Step 3] --> O([Output])
    G -->|fail| X([Reject])

    style I fill:#1d3557,color:#fff
    style O fill:#2d6a4f,color:#fff
    style X fill:#9d0208,color:#fff

The Gate is a plain code check, not a model call. Passing it carries the work forward; failing it ends the run at Reject — and that terminal is the point of the whole picture: the chain is allowed to stop, and stopping is a better outcome than continuing on a bad step.

Mechanism. Each call gets a fresh, small context containing only what that step needs.

This is the pattern’s real advantage and it is rarely stated. Because context does not accumulate, you never pay the quadratic term from the growth formula above. And no step suffers positional degradation — the U-shaped recall curve where content in the middle of a long window is retrieved worst (Why quality degrades in long contexts). Step 3 is as sharp as step 1.

Here is the whole pattern as pseudocode. Read the first line as a definition — it names the object the failure mode below turns on — and then the four lines under it as the chain proper.

spec    = the original request the chain was handed, as a structured object —
          spec.text is the prose, spec.declared_fields the field names it declares
draft   = llm(f"outline this spec: {spec.text}")
if not valid(draft, spec): reject()
code    = llm(f"implement: {draft}")
tests   = llm(f"write tests for: {code}")

Note what valid is given: both the draft and spec. That second argument is the whole subject of the failure mode below.

Use when: the task decomposes into steps you can name at design time — translate and then check the localization, or outline then draft then polish, or extract then validate then route.

Failure mode — error compounding. Step 1 hallucinates a field name — that is, states something no source supports, writing user_id when the spec says customer_id.

Steps 2 and 3 then build faithfully on top of it. The output is internally consistent and entirely wrong, and nothing in the chain can notice, because each step only sees its predecessor’s output and never the original source.

The fix is the gate, and gates must check against the original input, not the previous step.

Two names appear in the code below, and both come from the pseudocode above. spec is the chain’s original request as the structured object declared there, so spec.declared_fields is the set of field names the spec declares. fields(draft) is a small helper of yours that pulls out the set of field names the draft actually used.

# ✗ checks self-consistency — passes on a compounded error
if valid_json(draft): ...

# ✓ checks against ground truth. `<=` between two Python sets is the subset
#   test: it is True only when every element on the left is also on the right.
#   So this passes only if every field the draft used was declared in the
#   spec — one invented `user_id` and the gate rejects.
if fields(draft) <= spec.declared_fields: ...

That one operator is the entire content of the line. valid_json asks whether the draft is well-formed. The subset test asks whether the draft is true of the source, which is the only question that catches a compounded error.

A failed gate rejects; it does not continue. That is what the Reject terminal in the diagram is for, and it is the one edge people delete first.

A chain that logs a gate failure and proceeds to step 2 has reintroduced exactly the compounding it was built to stop, now with a warning line nobody reads.

If a step is worth another attempt, retry that step with the gate’s specific complaint appended to its prompt — and cap the retries. An uncapped retry inside a chain silently converts a fully predictable N× into an unbounded one.

Cost. N× for N steps, each on a small context. Fully predictable — you can quote it to finance before writing a line.

Latency. This means the wall-clock time a user waits, and in a chain it is serial: N × TTFT + N × decode.

TTFT is time-to-first-token, the wait before the first output token appears. It is dominated by prefill, the single pass the model makes over the whole input before it can emit anything. Decode is the one-token-at-a-time generation that follows.

Both terms multiply by N because nothing overlaps: step 2’s prompt does not exist until step 1 has finished decoding.

A chain also runs every input through the same pipeline on the same model — the next pattern is what you add when the inputs stop being alike.

2. Routing

Most of a support queue does not need your best model, but something has to decide which requests do. A router — one cheap call that classifies the input and dispatches it to a handler built for that class — is usually the first cost lever to pull, ahead of rewriting any prompt, and the derivation below shows why.

The shape, in plain words. A router is a small model call that reads the incoming request, returns one label from a short list you defined, and hands the request to the branch wired to that label. Each branch is a different design: its own prompt, its own tools, and possibly its own model size.

The problem it solves. A single prompt covering every kind of input has to hedge across all of them, and hedging costs both accuracy and money. You end up running your most expensive model on requests a cheap one would have answered perfectly.

The diagram below names three models, so read this first. Throughout this chapter, Haiku, Sonnet and Opus are the small, mid and large tiers of the Claude model family — cheapest and fastest, through to most capable and most expensive.

They are not three products with different skills. They are one ladder, and choosing a rung is the lever this section is about. The concrete list prices arrive a few paragraphs down, where the cost is derived.

flowchart TD
    I([Input]) --> R{Router · Haiku}
    R -->|refund| A[Refund handler<br/>Haiku + tools]
    R -->|technical| B[Tech handler<br/>Opus + RAG]
    R -->|abuse| C[Escalate to human]
    R -->|unknown| D[Fallback: general handler]
    A --> O([Output])
    B --> O
    D --> O
    C --> H([Human queue])

    style I fill:#1d3557,color:#fff
    style R fill:#bc6c25,color:#fff
    style O fill:#2d6a4f,color:#fff
    style D fill:#40916c,color:#fff
    style H fill:#1d3557,color:#fff

The four branches are not four copies of the same thing.

• refund goes to a Haiku handler that has write tools and a narrow policy prompt.
• technical goes to Tech handler Opus + RAG — retrieval-augmented generation, defined above (chapter 05).
• abuse gets no handler at all. It is an escalate to human edge into a human queue. The load-bearing detail is that this branch never reaches Output, so no model ever composes a reply to an abusive message.
• unknown is the fallback: general handler — the branch that exists so the router is never forced to pick the least-wrong of k classes for an input that belongs to none of them.

Mechanism. A router buys two independent wins.

The first is specialization: each branch gets a prompt tuned for one job, instead of one prompt hedging across all of them.

The second is model tiering: the Router · Haiku box is a ~200-token classification on the cheapest model, and its label decides which price tier the expensive call runs at.

Use when: the input space splits into a small number of named classes that want different tools, not just different wording — and you can label a few hundred real inputs to check the split is real.

Failure mode — the silent misroute. The model routes a technical question to the refund branch. The refund handler has no retrieval tool, so it answers from prior knowledge, confidently and wrongly. Branches have no path back, so nothing corrects it.

Three guards:

1. Always include a fallback route. Never force a choice among k known classes.
2. Log the routing distribution. A class that never fires means your router or your taxonomy is broken. A class that fires 80% of the time means your taxonomy is wrong.
3. Let downstream handlers escalate back. A handler that finds itself with the wrong tools should be able to say so, not improvise.

The router in code

Here is the whole pattern against the Anthropic SDK, with the handlers stubbed. Two lines carry the design. The constrained instruction (“exactly one word”) makes the classifier’s output cheap and machine-checkable — but a model can still return Refund., an apology, or an empty string, so the label not in LABELS check is not optional: dispatching on an unvalidated string is a KeyError in production, and mapping anything unrecognized to unknown is the fallback branch doing its job. The dispatch itself is a plain dictionary, because once the label exists there is nothing left for a model to decide.

import anthropic

client = anthropic.Anthropic()

LABELS = {"refund", "technical", "abuse", "unknown"}

HANDLERS = {                                # plain Python — no model dispatches
    "refund":    lambda msg: f"[refund handler, Haiku + tools] {msg}",
    "technical": lambda msg: f"[tech handler, Opus + RAG] {msg}",
    "abuse":     lambda msg: "[human queue] escalated; no model composes a reply",
    "unknown":   lambda msg: f"[general fallback handler] {msg}",
}

def route(message: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",           # the cheapest rung does the classifying
        max_tokens=10,
        system="Classify the support message. Reply with exactly one word: "
               "refund, technical, abuse, or unknown. No punctuation.",
        messages=[{"role": "user", "content": message}],
    )
    text = next((b.text for b in resp.content if b.type == "text"), "")
    label = text.strip().lower()
    if label not in LABELS:                 # never dispatch on an unvalidated string
        label = "unknown"
    handler = HANDLERS[label]
    return handler(message)

print(route("I was charged twice for order 4021"))

Cost, derived. With traffic shares sᵢ routed to models costing cᵢ:

cost = c_router + Σ sᵢ·cᵢ

Read that as: the router’s own cost, plus, for each branch i, the share of traffic that lands there multiplied by what that branch costs to run.

Now substitute real prices. Prices are quoted per MTok — one million tokens — and always as a pair, input rate then output rate, because generated tokens cost several times more than read ones. This is the only price table in the chapter; every dollar figure anywhere below is substituted from it.

These are list prices as of mid-2026, for the model generation this file writes against — treat the date as part of the number. Vendors reprice and ship new tiers, so a figure quoted from memory a year from now will be wrong, and quoting a stale one confidently is worse than saying you would look it up.

The habit worth keeping is not the three rows but the ratios they encode: output is 5× input on every rung ($5/$1, $15/$3, $25/$5), and the small-to-large spread is also 5× ($5/$1 on input, $25/$5 on output). Those survive a price change; the absolute numbers do not.

Now price one request. Take a 200-token router prompt returning ~5 tokens, and a handler call of 2,000 tokens in / 400 tokens out. Each line below multiplies a token count by its per-MTok rate — so 200 · $1/MTok is 200 × $1 / 1,000,000 = $0.0002.

router   (Haiku)   200·$1/MTok  +   5·$5/MTok   = $0.000225
handler  (Haiku)  2000·$1/MTok  + 400·$5/MTok   = $0.0040
handler  (Opus)   2000·$5/MTok  + 400·$25/MTok  = $0.0200

router / Opus handler = 0.000225 / 0.0200 = 1.1%      ← the "rounding error"
Haiku / Opus handler  = 0.0040   / 0.0200 = 0.20      ← the tiering factor

Now blend the branches. Suppose 60% of traffic can be answered on Haiku and 40% needs Opus. Substitute into cost = c_router + Σ sᵢ·cᵢ:

0.000225  +  0.6 · $0.0040  +  0.4 · $0.0200
0.000225  +     $0.0024     +     $0.0080     =  $0.010625

Against $0.0200 for sending everything to Opus, that is 0.010625 / 0.0200 = 0.53×, roughly half.

The router itself accounts for 0.01 of that 0.53. Drop the router term and you get the clean 0.6·0.2 + 0.4·1.0 = 0.52. The difference between 0.52 and 0.53 is the router, which is the point: the classification call is a rounding error against the savings it enables.

That is why routing is usually the first cost lever, ahead of prompt optimization. Case study 06 prices the same structure end to end and measures it by leave-one-out: turn off one optimization at a time, keep everything else on, and attribute to it whatever the bill rises by.

By that method, routing alone is worth $0.094 → $0.034 per ticket, a 2.8× reduction on the model bill. Routing plus tiering plus caching together is $0.560 → $0.034, or 16.5×.

State caching’s own leave-one-out number too, or the ordering claim cannot be checked. The same table prices it the same way: switch caching off in the otherwise-finished design and the ticket goes $0.105 → $0.034, which is 3.1× against routing’s 2.8×.

So caching does edge out routing in raw dollars, and the honest thing is to say so rather than overclaim for the lever you are recommending. Routing still goes first, for a reason that is not about dollars: you cannot run 60% of traffic on Haiku until something has decided which 60%.

3. Parallelization

“Just run it in parallel” names two different designs — one splits the work, the other splits the risk — and in both, the real engineering sits in the combiner at the bottom of the diagram, not the fan-out at the top.

The shape, in plain words. Parallelization issues several model calls at the same time from one input and combines their answers into one. Two versions of that get conflated even though they solve different problems.

Sectioning splits the work: each call sees a different slice of the input, and the combiner stitches the slices back together.

Voting splits the risk: every call sees the same input, and the combiner compares the answers before deciding what to trust.

The diagram puts the two side by side. They have the same fan-out shape, so look at the labels on the boxes, not the arrows.

flowchart LR
    subgraph SEC["Sectioning — split the work"]
        I1([Input]) --> S1[Chunk A]
        I1 --> S2[Chunk B]
        I1 --> S3[Chunk C]
        S1 --> AGG1[Merge]
        S2 --> AGG1
        S3 --> AGG1
    end

    subgraph VOT["Voting — split the risk"]
        I2([Input]) --> V1[Lens 1]
        I2 --> V2[Lens 2]
        I2 --> V3[Lens 3]
        V1 --> AGG2[K-of-N]
        V2 --> AGG2
        V3 --> AGG2
    end

The two subgraphs share a shape and differ only at the bottom: Merge stitches three different slices back together, while K-of-N compares three answers to the same input. The aggregator is where the difference actually lives.

Sectioning mechanism. Sectioning splits an input too large for one context, or a set of subtasks that never need to see each other. Each worker gets a small window — so, again, no quadratic term and no positional decay.

Voting mechanism, and the part people get wrong. Running the same prompt N times reduces variance — the run-to-run randomness that comes from the model sampling its next token rather than always taking the most likely one (Sampling and why temperature0 isnt deterministic).

But it does nothing about systematic error, the kind where the model is wrong the same way every time. If the model misreads the question, all N runs misread it identically. You get three confident wrong answers that agree, which reads as high confidence and is the opposite.

Variance is not the same as bias, and bias is the consistent lean in one wrong direction. To attack bias you need diverse lenses: N different prompts attacking the problem from different angles — correctness, security, performance, “does this reproduce” — not N identical samples.

Redundancy catches flakiness. Diversity catches blind spots.

The aggregators are the hard part, not the fan-out

Merge is a design decision with a cost attached, and which design you pick depends on whether the chunk outputs overlap.

If they never overlap — findings listed per file, rows returned per database partition — merge is a concatenation: you join the pieces end to end in code, and it costs nothing.

If they overlap, contradict, or need to become one narrative, merge is another LLM call over all N outputs. That call is not included in the N× figure below, and it reintroduces a single long context at the end of a pattern you chose in order to avoid long contexts.

Size it before you assume it is free. Five chunk outputs at 800 tokens each is a 5 × 800 = 4,000-token merge prompt, which is trivial. Forty at 800 is 40 × 800 = 32,000 tokens, and the merge is now the most expensive call in the design.

K-of-N is a threshold you choose, not a property of the pattern. It says how many of the N lenses must agree before the system acts. On the correctness / security / performance lenses above:

• 1-of-3 means “escalate if any lens objects”. It catches a high share of the real problems and raises a lot of false alarms — the right trade for a security gate.
• 3-of-3 means “ship only on unanimous pass”. Almost everything it does ship is genuinely fine, at the price of blocking constantly.
• 2-of-3 is the usual compromise.

The trap is what you do with the minority. Majority-voting a 2–1 split discards the dissent, and on diverse lenses the dissent is the entire product. The security lens objecting alone is not noise to be outvoted; it is the finding.

So let K decide only whether to block, never whether to report. Surface every objection along with the lens that raised it. K-of-N as a silencer is defensible only in the redundancy case, where all N ran the same prompt and disagreement really is sampling noise.

Use when: reach for sectioning when the input exceeds the context window defined at the top of this chapter, or when the subtasks are genuinely independent. Reach for voting when a single sample’s run-to-run randomness is the risk you are buying down, and you can afford N× tokens to do it.

Failure mode — lost cross-chunk context (sectioning). A bug spanning two files is invisible to both workers, because neither sees the other’s file. Three mitigations: overlap the sections, give every worker a shared summary of the whole, or add a final pass that only looks at the boundaries.

Failure mode — correlated agreement (voting). This is what the Voting — split the risk label quietly over-promises, because N identical samples split sampling risk and nothing else.

Three runs of the same prompt on a misread question return three matching wrong answers, and the aggregator reports 3-of-3 — the highest-confidence output the system can emit, on its worst input. That is worse than a single call, which at least would not have claimed consensus.

Detection: track the agreement rate per input slice. A lens set that agrees ~100% of the time is not three lenses, it is one lens run three times, and you are paying 3× for it. The fix is diversity in the prompts, not more samples.

Voting in code

The example below fans one diff across the three lenses on real threads and combines the verdicts in plain Python — no fourth model call sits at the bottom, because counting objections is arithmetic, not judgement. The PASS sentinel is what makes the combine step code: each lens either returns the exact string or an objection, so K-of-N reduces to a dictionary comprehension and a length check. And note the loop prints every objection before the block decision — K decides whether to block, never whether to report, which is the minority-report rule from above enforced in four lines.

import anthropic
from concurrent.futures import ThreadPoolExecutor

client = anthropic.Anthropic()

LENSES = {
    "correctness": "Review the diff for logic errors only.",
    "security":    "Review the diff for security problems only.",
    "performance": "Review the diff for performance regressions only.",
}

def review(lens: str, instruction: str, diff: str) -> tuple[str, str]:
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=instruction + " If you find nothing, reply with exactly: PASS",
        messages=[{"role": "user", "content": diff}],
    )
    text = next((b.text for b in resp.content if b.type == "text"), "")
    return lens, text.strip()

diff = open("change.diff").read()
with ThreadPoolExecutor(max_workers=3) as pool:
    futures = [pool.submit(review, lens, inst, diff) for lens, inst in LENSES.items()]
    results = [f.result() for f in futures]

objections = {lens: text for lens, text in results if text != "PASS"}
for lens, text in objections.items():       # report every objection, always
    print(f"[{lens}] {text}")
print("BLOCKED" if objections else "PASS")  # K-of-N with K=1: any lens blocks

Swap the three lens prompts for three chunks of one long input and the same skeleton is sectioning — the only structural change is that the combine step becomes a concatenation of the results instead of a threshold over them.

Cost. N× tokens. But wall-clock time — what the user actually waits — stays ~1×, because the calls run at the same time. Add the merge call on top if merge is itself an LLM call.

This is the cheapest latency win in the catalog, and it is the case where cost and latency move in opposite directions, so quote both numbers rather than one.

Everything so far assumed you could enumerate the pieces yourself; the next pattern is what you build when only the model can.

4. Orchestrator–Worker

One manager call splits a task into however many pieces this particular input needs, each piece runs in a worker with its own blank context, and a final call combines what they return. That is orchestrator–worker — and the argument for it is capability, not speed.

The shape, in plain words. It is sectioning (Parallelization) with one thing changed: the model decides how many workers there are and what each one does.

That is exactly the line between a workflow and a model-directed pattern. But it is a bounded crossing, which is why the tree files this under bounded agency rather than as a true agent: the model picks the fan-out, while you fixed the shape (decompose → workers → synthesize) and the ceiling on how many.

The problem it solves. Some tasks require reading far more material than fits in one conversation, and the only way to read it all is to read it in separate conversations that never meet.

The diagram shows one task becoming N independent worker runs and then one synthesis. Notice the two annotations on the arrows — the window each worker gets, and the much smaller summary it returns.

flowchart TD
    I([Task]) --> O[Orchestrator<br/>decomposes]
    O --> W1["Worker 1<br/>own 60k window"]
    O --> W2["Worker 2<br/>own 60k window"]
    O --> W3["Worker N<br/>own 60k window"]
    W1 -->|~800-token summary| S[Synthesizer]
    W2 -->|~800-token summary| S
    W3 -->|~800-token summary| S
    S --> F([Result])

    style I fill:#1d3557,color:#fff
    style O fill:#40916c,color:#fff
    style F fill:#2d6a4f,color:#fff

Mechanism — context isolation, not parallelism. This is the sentence to say. Walk the boxes in order.

Orchestrator decomposes turns one task into w briefs, and it is the only call that sees the whole problem.

Each of Worker 1, Worker 2, … Worker N then runs in its own window of ~60k tokens — its own conversation, its own tool results, starting from nothing but its brief. That isolation is why the workers cannot poison each other’s context, or the orchestrator’s.

Each worker returns an ~800-token summary to the Synthesizer. The synthesizer is a different call from the orchestrator, precisely because it should judge the returned evidence rather than defend the plan that produced it.

The 60k and the ~800 are an illustrative configuration, not measured constants. They are a worked example chosen because it is roughly what a research fan-out looks like in practice. 60k is a window a worker can fill with tool output without running into the ceiling or the worst of the recall curve; 800 tokens is about the length of a report that states findings with citations and nothing else.

Both numbers are yours to set, and both are levers the cost derivation below prices explicitly. Substitute your own and the shape of the argument does not change — which is why it is written as a ratio:

6 workers × 60k explored   = 360k tokens read
6 summaries × 800          =   4.8k tokens the orchestrator holds
                              ~75× compression

The compression figure is 360,000 / 4,800 = 75. So the orchestrator gets the benefit of 360k tokens of reading while only ever holding 4.8k of it.

A single agent physically cannot read 360k tokens of source material and still have room to reason about it. That is the argument, and it is a capability argument, not a speed one.

Use when: the decomposition is real but how many pieces it has depends on the input — “audit every service that touches billing” when you don’t know how many that is, or “research this claim across whatever sources exist”. If you can write down the number of pieces at design time, use sectioning instead and keep the cheaper, more debuggable pattern.

Failure mode — worker conflict. Two workers edit the same file, and the second overwrites the first. Or two return contradictory findings and the synthesizer averages them into something true of neither.

Here is the second failure as a trace. Read the two worker lines, then the synthesis line, and notice that the synthesis is not wrong so much as it is empty:

worker_1: "The rate limit is 100 req/min [docs/api-v2]"
worker_2: "The rate limit is 1000 req/min [blog/scaling-2023]"
synthesis: "The rate limit is in the hundreds of requests per minute."   ← useless

Two fixes, both structural:

1. Disjoint scopes in the brief — no two workers may touch the same file or entity.
2. A synthesizer instructed to surface conflicts, along with where each finding came from, rather than reconcile them.

The fan-out in code

The pattern’s whole structure fits in one page: an Opus call emits the decomposition as JSON, threads run the workers, an Opus call synthesizes. Three lines are load-bearing. json.loads(plan) is the contract check — if the orchestrator wraps its array in prose or code fences, the right response is to fail loudly and re-ask, not to parse harder, because a planner that cannot follow its own output format should not be trusted with the plan. The [:6] cap is the “bounded” in bounded agency: the model picks the fan-out, your code holds the ceiling. And worker receives only its brief — that argument list is the context isolation the whole section argues for.

import anthropic, json
from concurrent.futures import ThreadPoolExecutor

client = anthropic.Anthropic()

def ask(model: str, system: str, user: str) -> str:
    resp = client.messages.create(model=model, max_tokens=1024, system=system,
                                  messages=[{"role": "user", "content": user}])
    return next((b.text for b in resp.content if b.type == "text"), "")

task = "Find the documented rate limits of our three upstream APIs."

plan = ask("claude-opus-5",                 # the one call that sees the whole problem
           'Decompose the task into independent research briefs. Reply with a '
           'JSON array of strings, e.g. ["...", "..."], and nothing else.',
           task)
briefs = json.loads(plan)[:6]               # contract check + a cap the model cannot move

def worker(brief: str) -> str:              # fresh context: the brief and nothing else
    return ask("claude-haiku-4-5",
               "Answer the brief in under 150 words. Findings and sources only.",
               brief)

with ThreadPoolExecutor(max_workers=len(briefs)) as pool:
    summaries = list(pool.map(worker, briefs))

print(ask("claude-opus-5",                  # a separate synthesis call, not the planner
          "Synthesize the worker reports. Surface conflicts with their sources; "
          "do not average them.",
          "\n\n".join(f"[worker {i}] {s}" for i, s in enumerate(summaries))))

Real workers are small tool-using loops rather than single calls — this skeleton stubs them to one call each so the wiring stays visible.

Cost, derived. The multiplier is built from parts you can count, not quoted from folklore. Each worker runs its own small agent loop, so it pays the same quadratic growth as any loop — but only over its own short conversation, and that is the term that makes the whole thing affordable:

orchestrator:  1 decompose + w summaries in + 1 synthesis
each worker:   s steps, quadratic in s, over its own prefix
total ≈ w · (worker loop cost) + orchestration overhead

That is a formula with symbols in it, so substitute, or 4–15× is a number you memorized. Four quantities, and every one of them is something you either choose or measure:

• w — how many workers the orchestrator spawns. Take 4 at the low end, 6 at the high end.
• W — what one worker bills across its whole run: the tokens it reads and writes over its s steps, quadratic in s, but over its own short prefix. Take 40,000 at the low end and 60,000 at the high end, matching the window in the diagram.
• O — the orchestrator’s own bill. It pays its P = 6,000-token prefix to decompose, reads w reports of ~800 tokens each, then pays both again to synthesize. So O = 2 · (6,000 + w·800). Charging the decompose call for reports it has not received yet is a deliberate over-estimate; it makes O an upper bound and the conclusion survives the padding.
• B — the denominator, and naming it is the whole trick. 4–15× is a ratio, and a ratio with an unstated denominator is not a number. B is what a single agent would have billed on the same question: one context, n·P + a·n²/2, running until it decided it had read enough and stopping there. Take 40,000 for a single agent that explored properly and 25,000 for a thriftier one that stopped sooner.

The multiplier is (w·W + O) / B. The table below substitutes two configurations, one at each end of the usual range. Read across a row: workers plus orchestrator gives the total, and the total divided by B gives the multiplier.

The two divisions in full: 178,400 / 40,000 = 4.46 at one end, and 381,600 / 25,000 = 15.26 at the other.

That is where the range comes from: both sides of the ratio move at once. The high end has more workers reading deeper windows and is compared against a stingier single agent. Neither end is a worst case; they are the two ends of the configuration people actually build.

(Deriving the numbers and chapter 06 derive the same two rows from the same four quantities. If your own substitution lands somewhere else, one of the four inputs is different, and that is the thing to name.)

Now notice how small O is. It is 18,400 / 178,400 = 10.3% of the low-end total and 21,600 / 381,600 = 5.7% of the high-end one — so 5.7–10.3% of the bill.

The orchestrator is not the expense. w × W is the expense, and that tells you which levers matter:

• Worker count w is linear — double the workers, double that term.
• Worker step budget s is quadratic within each worker.
• Worker model tier moves the dominant term.
• Downgrading the orchestrator saves almost nothing, because it was a tenth of the total to begin with.

Price the tier lever, since it is the one people quote loosely. Move the workers to Sonnet and keep the orchestrator and synthesizer on Opus. The worker share then bills at $3/$15 instead of $5/$25, which is 3/5 = 0.6× on input and 15/25 = 0.6× on output.

So in a design where workers are 90% of the tokens, the overall factor is 0.9 × 0.6 + 0.1 × 1.0 = 0.64×, not half. Quote 0.64, not “roughly half”; an interviewer who knows the price list will notice.

Read the 4–15× multiplier correctly: it measures work volume, not inefficiency. You are comparing one agent that stopped at 8 searches against 4 workers doing 8 each.

Per unit of work, fan-out is cheaper: w workers pay a·n²/(2w), so the quadratic term divides by the worker count (ch 06). Fan out to buy more reading, not to make reading cheaper.

5. Evaluator–Optimizer

A draft-and-critique loop sets two calls against each other: one generates, a second judges the result against criteria you wrote down, and the generator revises until the judge passes it or the round cap stops it. The load-bearing detail is that the judge runs as a separate call with a fresh context — it never sees the conversation that produced the draft.

The shape, in plain words. Generate, critique, revise, repeat.

The problem it solves. A first draft is usually close but wrong in specific, nameable ways. A model handed those specific complaints fixes them reliably, where the same model asked to “do better” simply produces a different first draft.

The catch is that this only works when you can articulate the criteria, which is exactly the condition on the tree branch that led here.

The diagram has one loop and three exits. Look at the labels on the edges leaving the evaluator: two of them reach Output, and only one of those is a success.

flowchart LR
    I([Task]) --> G[Generator]
    G --> E{Evaluator<br/>fresh context}
    E -->|pass| O([Output])
    E -->|fail + structured feedback| G
    E -.->|max rounds| O

    style I fill:#1d3557,color:#fff
    style O fill:#2d6a4f,color:#fff
    style E fill:#bc6c25,color:#fff

Mechanism — why the fresh context matters. If you ask the same conversation “is this good?”, the draft is already in its context, and the model is now predicting the continuation of a conversation in which it just produced that draft.

Two forces push it toward approval. Sycophancy is the model’s trained tendency to agree with whatever is in front of it rather than contradict it. Positional recency is the fact that the most recent text in a context carries outsized weight on the next token.

A separate call sees the draft as input to judge, not as something I just wrote. This is the same reason a code reviewer is not the author.

The edges out of the evaluator are the whole control flow. There are three of them.

On pass, the draft leaves and the run is done.

On fail, the loop returns to the Generator — but the edge is labelled fail + structured feedback for a reason. What goes back is not the verdict but the specific unmet criteria. A generator told only “rejected” has nothing to change, so it resamples rather than revises.

The dotted max rounds edge is the one people forget to build. When the round cap is reached, the loop exits to the output anyway, carrying the best draft and a passed=False flag.

Both pass and max rounds reach Output. Only one of them is a success, and a system that cannot tell you which one it took is worse than no evaluator at all.

Use when: you have clear, articulable criteria and iteration measurably helps — literary translation judged against a style guide, code that must pass a linter, or a research answer that has to cover N named sources.

Failure mode — oscillation. With vague criteria, round 3 undoes round 2. The trace below shows four rounds; watch the draft return to where it started while the bill keeps going up:

r1: draft mentions no caveats
r2: judge says "add caveats"        → draft adds three paragraphs of caveats
r3: judge says "too hedged"          → draft removes all caveats
r4: judge says "add caveats"         → ...

Three fixes:

1. Cap the rounds.
2. Make the evaluator return structured per-criterion verdicts rather than prose, so “too hedged” cannot silently replace “add caveats”.
3. If a criterion can be checked in code, check it in code. Whether the tests pass, whether the JSON parses, and whether a required section exists are all questions a program answers exactly, every time. An LLM judge is the fallback for genuinely subjective dimensions, not the default.

Cost. 2N× for N rounds — strictly 2N+1 calls, since the first draft is generated before the loop begins. So a 3-round cap is 2×3 + 1 = 7 calls, not 6. Immaterial at N≥2, but say it if asked. Empirically rounds 1–2 capture most of the gain; beyond 3 you are usually paying for churn.

Cache the judge’s system prompt. Prompt caching means the provider stores its precomputed internal state for the leading, unchanged part of your prompt, and bills a repeat of that part at roughly a tenth of the normal input rate (Prompt caching derived).

The judge’s system prompt is byte-identical every round, so rounds 2 and later read the criteria at ~0.1× input cost.

One caveat makes this advice safe rather than aspirational: the cached prefix has to clear the model’s minimum, which is 512 tokens on claude-opus-5. Below that, nothing caches. There is no error — the only symptom is cache_creation_input_tokens: 0 in the usage block the API returns alongside the reply.

A terse judge prompt plus three bullet criteria will not reach 512 tokens. The full criteria list that makes the pattern work usually will.

This is the one pattern in the chapter you are likely to be asked to write out, so it gets a file of its own. 02a — Building One: Evaluator–Optimizer, Three Tiers codes the loop at three levels of fidelity: whiteboard pseudocode, a LangGraph version where the retry is a graph edge rather than a while, and a production version against the Anthropic SDK with the criteria written out and the judge’s prompt cached.

It also shows which criteria to move out of the judge and into ordinary code, which is the mechanical version of the “check it in code” advice above.

6. ReAct — Reason + Act

The model reasons about what it needs, calls a tool, reads the result, and repeats until it decides it is finished — that loop is what most people mean by “agent”. Below it is written out as a real message array, along with the two ways it fails and why its cost grows the way it does.

The shape, in plain words. ReAct interleaves three things inside one growing conversation:

• a thought about what is needed next,
• an action that calls a tool,
• an observation, which is that tool’s result fed back in.

Then it thinks again with the result in hand, and repeats.

The problem it solves. Some tasks cannot be planned in advance, because what you do third depends on what the second step returned. ReAct refuses to decide anything before it has to.

flowchart TD
    S([Goal]) --> T[Thought<br/>what do I need?]
    T --> A[Action<br/>tool call]
    A --> OB[Observation<br/>tool result]
    OB --> D{Enough?}
    D -->|no| T
    D -->|yes| F([Answer])

    style S fill:#1d3557,color:#fff
    style F fill:#2d6a4f,color:#fff

The division of labour in the diagram matters: the model produces the thought and the action, and your code runs the tool and returns the observation. Every lap appends to the same conversation, and that one fact is what the cost derivation below turns on.

Mechanism, modern form. You do not hand-prompt Thought:/Action:/Observation: any more. Native tool calling is ReAct. The three parts map onto the API like this:

• the thought is the model’s reasoning, emitted as ordinary text;
• the action is a tool_use block, a structured request to run one of your tools;
• the observation is a tool_result block you send back.

The tool_use block is emitted under constrained decoding, meaning the runtime blocks any token that would break the tool’s declared shape, so the call always parses (Structured output is a guarantee not a request).

Know where the name comes from. The paper is ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al., 2022 (arXiv:2210.03629). Its contribution was the observation that interleaving reasoning traces with actions beats doing either alone.

What the paper proposed as a prompt format — the model made to literally emit the strings Thought:, Action: and Observation: — the API now provides as a message structure. The idea survived; the string formatting did not. Saying that shows you know both the paper and the current API.

Here is the mapping in a real message array. A message array is the list of alternating turns you resend on every request. This one is a single turn of debugging “the checkout endpoint 500s on empty carts”.

The diagram’s first two boxes, Thought what do I need? and Action tool call, arrive as one assistant message holding two content blocks — so look at how one resp.content list carries both:

resp.content == [
    TextBlock(text="The 500 is probably in the totals path. Let me read the handler."),   # Thought
    ToolUseBlock(id="toolu_01A", name="read_file",                                        # Action
                 input={"path": "api/checkout.py", "lines": "40-80"}),
]

The diagram’s Observation — tool result is what you append on the next request, and it is a user-role message.

Roles are how the API labels who produced each turn: assistant for the model’s own output, user for everything you send in. The environment has no role of its own, so tool results are sent under the user role. In the code below, note that the assistant message is appended back verbatim — both blocks, unedited — and that tool_use_id is what ties the result to the request that asked for it:

messages.append({"role": "assistant", "content": resp.content})      # BOTH blocks, verbatim
messages.append({"role": "user", "content": [
    {"type": "tool_result", "tool_use_id": "toolu_01A",              # Observation
     "content": "43: total = sum(i.price for i in cart.items)\n44: return total / len(cart.items)"},
]})

And the Enough? diamond is not a node you write. It is resp.stop_reason on the next response: "tool_use" is the no edge back to Thought, and "end_turn" is the yes edge to Answer.

That is the entire loop. There is no ReAct library; there is a while on a stop reason. A worked trace walks the same array end to end, including the five ways people break it, and The tool calling loop builds the loop as a runnable exercise — so it is not duplicated here.

Use when: the number and order of steps genuinely depend on what the environment returns — debugging, search, and anything else where step 3 is unknowable until step 2 has run.

Failure mode 1 — the loop

The agent calls the same tool with the same arguments three turns running.

The mechanism is worth stating precisely, because it explains the fix. The model’s context now contains three near-identical (call, result) pairs. A model predicting the next token from a context full of a repeated pattern makes the fourth repetition more likely, not less. The loop reinforces itself.

Detection and the break are in Infinite loops. The key move is to feed the trip message back before halting.

A trip message is the text your loop-detector writes into the conversation at the moment it trips — the moment your code notices the repetition. It is not an error and not a halt. It is a plain observation addressed to the model, along the lines of:

you have called search_docs with the query rate limit three times and received the same result each time; that approach is not working, try something else.

You append it exactly where a tool result would have gone, and let the loop take one more lap.

The reason to bother is mechanical. The repetition is self-reinforcing only because the context contains a pattern and nothing contradicting it, so inserting the contradiction is the cheapest thing that can break it. Told explicitly that it is repeating, the model usually changes strategy. Halt only if the next lap repeats anyway.

Failure mode 2 — context blowup

Every observation is appended forever, so turn n sends P + a·n input tokens — where P is the system prompt plus tool schemas, and a is what one (call, result) pair adds.

Per-turn cost therefore grows linearly in n. It is the running total that is quadratic, at n·P + a·n²/2 (Deriving the numbers).

Calling the per-turn curve quadratic is the usual slip, and it matters because it changes what the fix is. Trimming old observations attacks a, which shrinks both terms. The only lever on n² is capping the steps — and raising the cap moves the wrong way, since a larger n is precisely what makes the quadratic term bite.

Substitute a 2k-token base prompt and 1.5k per step:

turn  2   2,000 +  2 · 1,500 =  5,000 tokens
turn 20   2,000 + 20 · 1,500 = 32,000 tokens
                              32,000 / 5,000 = 6.4×

So the growth from turn 2 to turn 20 is 6.4×, not the often-quoted 10×. The 10× figure is the limit as the base prompt becomes negligible against accumulated history — 20a / 2a = 10 with P dropped entirely. With a real system prompt in the mix you land below it.

Either way, the second half of this failure is worse than the bill. At turn 20 the instructions sit in the middle of a 32k window, the worst position on the recall curve (Why quality degrades in long contexts). That is why long ReAct sessions drift.

Cost. 5–20×, unbounded without a cap. This is the pattern that needs guards most, and the only one where “add a step cap” is genuinely non-negotiable rather than good hygiene.

7. Plan-and-Execute

Separate deciding from doing and the bill changes shape: one expensive call writes the entire plan up front, cheap calls carry out the steps one by one, and the plan is rewritten only when a step fails. The gap between this and ReAct is not a fixed multiple — it widens with the length of the task, and the derivation below shows how fast.

The shape, in plain words. Look around briefly, plan the whole thing once with a strong model, then execute the steps with a cheaper one, and replan only on failure.

The problem it solves. ReAct pays for its strongest model to make every decision, inside a conversation that keeps growing — when in fact most steps are simple instruction-following. Deciding is hard and happens once. Following a written instruction is easy and happens n times.

In the diagram below, the boxes are colour-coded by model tier: the two green boxes near the top run on the strong model, and everything after the queue runs on a cheap one. Also look at the three ways out of Step OK?.

flowchart TD
    G([Goal]) --> RECON[Recon: 1-2 cheap<br/>reads of the environment]
    RECON --> P[Planner<br/>Opus, once]
    P --> Q[[Step queue]]
    Q --> E[Executor<br/>Haiku/Sonnet, per step]
    E --> C{Step OK?}
    C -->|yes, more left| Q
    C -->|yes, done| F([Result])
    C -->|no| RP{Replans < 3?}
    RP -->|yes| P
    RP -->|no| X([Fail loudly])

    style G fill:#1d3557,color:#fff
    style RECON fill:#40916c,color:#fff
    style P fill:#40916c,color:#fff
    style F fill:#2d6a4f,color:#fff
    style X fill:#9d0208,color:#fff

Walk the boxes, because three of them carry the whole cost argument.

Recon: 1-2 cheap reads of the environment is reconnaissance — a couple of cheap look-first calls, such as listing the directory or reading the schema (the declared shape of the data: its tables and their columns). Their only job is to make sure the plan is written against reality rather than against a guess.

Planner Opus, once says the strong model is invoked exactly one time per plan, not once per step. That is where the intelligence budget goes.

Step queue is a real, inspectable data structure — a list you can print, diff, and hand to a human before a single side effect happens. That is the reviewability the comparison table below scores.

Executor Haiku/Sonnet, per step is the cheap tier, and it is cheap because the queue already made the decisions.

Step OK? then fans three ways, and all three matter. yes, more left pops the next item and keeps the same plan. yes, done exits with a result. no goes to the replan counter.

Replans < 3? is a hard stop. Its no edge goes to fail loudly — a terminal that reports the partial state and the last error rather than returning something plausible. An agent that quietly returns a half-done result on plan exhaustion is indistinguishable from one that succeeded, which is the worst property a long-running system can have.

Mechanism — why it is cheaper than ReAct over the same number of steps. Two reasons, both from Deriving the numbers.

First, each executor step runs on a short context — the plan plus this one step — so nothing accumulates and there is no quadratic growth.

Second, execution runs on a smaller model, because following an explicit instruction is a much easier task than deciding what to do next.

Use when: the horizon is long — meaning many steps separate the start from the finish — the environment is stable enough that a plan written now is still true in ten steps, and a human may want to read the plan before anything executes. That last one is often the real reason.

Failure mode — the stale plan. The plan was written before the environment was understood, so step 4 assumes a file that does not exist.

The recon phase exists precisely for this: one or two cheap reads before planning, so the plan is grounded. And cap replans at 3 — a third failed plan means the goal is wrong, not the plan, and continuing burns budget on a misconception.

Cost, derived — and the gap is not a fixed multiple. Both patterns take the same base prompt P, the same per-step payload a, and the same n steps. Only the accumulation differs. Each line below is one contribution to the total input tokens:

ReAct              n·P + a·n²/2                    one context, grows
Plan-and-Execute   r·P                             r recon reads
                 + (P + r·a)                       one planner call, sees recon
                 + n·(p + a)                       n executor calls, plan p + this step

Substitute P = 2,000, a = 1,500, r = 2 recon reads, and p = 800 tokens of plan text. Plan-and-Execute’s three lines collapse to a single expression first:

P&E  =  r·P     + (P + r·a)     + n·(p + a)
     =  2·2,000 + (2,000 + 3,000) + n·2,300
     =  4,000   +     5,000       + n·2,300
     =  9,000   + 2,300·n

ReAct has no such collapse — its a·n²/2 term has to be evaluated per n. Here are both at three horizons. P&E is Plan-and-Execute, and the last column is how many times more input tokens ReAct reads:

n =  5   ReAct  28,750   P&E  20,500   →  1.4×
n = 12   ReAct 132,000   P&E  36,600   →  3.6×
n = 25   ReAct 518,750   P&E  66,500   →  7.8×

Check one row by hand. At n = 12, ReAct is 12·2,000 + 1,500·144/2 = 24,000 + 108,000 = 132,000, and Plan-and-Execute is 9,000 + 2,300·12 = 36,600. The ratio is 132,000 / 36,600 = 3.6.

Read the shape, not the numbers. ReAct’s term is quadratic in n and Plan-and-Execute’s is linear, so the ratio between them grows with the horizon rather than sitting at some fixed multiple. At five steps the patterns are within 40% of each other and the extra machinery is not worth it. At twenty-five, ReAct is reading half a million tokens to do the same work.

Do not confuse these three numbers with the two in the summary table — they have different denominators.

The 1.4×, 3.6× and 7.8× above measure ReAct against Plan-and-Execute: same task, same n, two patterns, one divided by the other.

The table at the end of the chapter measures each pattern against one plain call on the same task, which is why it reports 5–20× for ReAct and 3–8× for Plan-and-Execute. Those table entries are ranges for the ordinary reason that n varies between deployments — not because the gap between the two patterns widens. That widening is a separate fact, and it is the one this section just derived.

Divide the two table ranges end for end — 5/3 = 1.7 and 20/8 = 2.5 — and you get 1.7× to 2.5×, a blunt summary of the same story the three rows above tell step by step.

Model tiering multiplies the gap. Of the 36,600 Plan-and-Execute tokens at n = 12, only the 9,000 recon-plus-planner tokens need the strong model; the other 27,600 executor tokens run on Sonnet. Price both at input rates:

ReAct, all Opus     132,000 · $5/MTok                   = $0.660
P&E, recon+planner    9,000 · $5/MTok  = $0.045
P&E, executors       27,600 · $3/MTok  = $0.083
P&E total                                               = $0.128
                                     $0.660 / $0.128    = 5.2×

So Plan-and-Execute is 5.2× cheaper in dollars against 3.6× in tokens, because the cheap tier absorbs the bulk. (Input tokens only; output is comparable on both and does not change the shape.)

ReAct vs. Plan-and-Execute, on the five axes that decide between them. The row to argue over in an interview is Human can review before execution: that is the one Plan-and-Execute wins outright, and it is often the reason the design is chosen at all.

Hybrid — plan coarsely, ReAct within each step — is usually the best interview answer, and it’s what most production coding agents actually do.

8. Reflexion

When an attempt fails, a separate call writes down why in one sentence, that sentence is stored, and every later attempt starts with it in hand. Two versions of that idea share the diagram below, and they have very different bills.

The shape, in plain words. Reflexion is the draft-and-critique loop of Evaluatoroptimizer plus memory that survives the attempt: the lesson outlives the retry that produced it.

The problem it solves. A plain retry loop re-learns the same fact every time. It discovers on attempt 2 exactly what it discovered on attempt 1 of the previous run, and pays for the discovery twice. Writing the fact down converts a failed attempt from wasted spend into something reusable.

The diagram below is a retry loop with one extra box. The box to look at is the cylinder — the Lesson store — and specifically which edge writes to it and which edge reads from it.

flowchart TD
    T([Task]) --> A[Attempt]
    A --> EV{Succeeded?}
    EV -->|yes| O([Done])
    EV -->|no| R[Reflect:<br/>why did it fail?]
    R --> M[(Lesson store)]
    M --> A

    style T fill:#1d3557,color:#fff
    style O fill:#2d6a4f,color:#fff
    style M fill:#7209b7,color:#fff

Mechanism. Walk the loop once, box by box.

Attempt runs the task.

Succeeded? is a check on the outcome of acting rather than on the thing produced: the tests passed, the query returned the right row, the form submitted. On yes the loop exits to Done without writing anything.

Only the no edge reaches Reflect: why did it fail?. That is a separate call whose job is not to fix the task but to name the cause in one falsifiable sentence. That sentence, and nothing else, is written to the Lesson store.

The arrow from the store back to Attempt is retrieval. The next attempt’s prompt is assembled as task + retrieved lessons, with the lessons injected into the system prompt ahead of the task, so they read as standing constraints rather than as advice the model may weigh against the user’s request.

Three mechanics that arrow hides, and you should state all three.

When the write fires. The store is written only on failure, and only after reflection — never on success. Otherwise it fills up with descriptions of things that already work.

What gets retrieved. The simple and usually correct answer is all of them, under a hard cap of a few dozen short lines. Searching such a small store by meaning mostly returns noise, and can silently drop the one lesson that would have mattered. Graduate to a search only once the cap starts biting.

Where they land. They go into the system prompt, before the task, in a fixed order. Otherwise the leading text of the prompt changes every run, and you lose prompt caching on the one block that never changes.

This is where Reflexion differs from Evaluator–Optimizer, and the difference is not “memory”. Evaluator–Optimizer critiques the thing produced — the draft, the patch, the answer — against criteria you wrote down. Reflexion reflects on the outcome of acting in an environment, so it needs a real attempt to have happened and failed. That is why its success check is a test run and not a rubric.

Two scopes — within-task and cross-task

The name is ambiguous, and the cost line depends on which one you mean. Two different architectures share this diagram.

Within-task Reflexion: the store lives for one task, holds a handful of lessons about this task’s environment, and is thrown away at the end. Cost is bounded by the retry cap, and nothing persists.

Cross-task Reflexion: the store outlives the task and is loaded by every future run. This is the version that pays for itself, and the version that charges rent forever. The cost line below is this one.

In an interview, say which you mean in the first sentence. Conflating them is how you end up defending a permanent context tax you did not intend to build.

Cross-task only pays off if lessons transfer — meaning a lesson learned on task A is still true and relevant on task B.

That holds when the tasks all come from the same distribution, meaning they are drawn from the same underlying population: one codebase, one schema, one API, one toolchain.

“Fix bugs in this repo” transfers, because the table names do not change between tickets, and neither does the CI matrix — the set of language versions the project’s automated build tests against, CI being continuous integration. “Answer arbitrary user questions” does not transfer, and a store built from it is pure overhead.

Failure mode — junk lessons. The store fills with unfalsifiable advice. The two marked ✗ below are the failure; the two marked ✓ are what a good lesson looks like:

✗ "Be more careful with edge cases."          unfalsifiable, pure token tax
✗ "Think step by step."                        already true, adds nothing
✓ "orders_v2 is canonical; orders is a stale view kept for BI."
✓ "The CI matrix runs Python 3.10 — no match statements in runtime code."

The good ones are specific, non-obvious from the code, and falsifiable — meaning you could run a check that proves the lesson wrong, which is exactly what “be more careful” can never be. (BI in the third line is business intelligence, the reporting stack.)

Here is one full cycle producing that third line, so the arrows in the diagram are not abstract. Read it as: attempt fails against a known-good number, reflection names the cause, the store gains one line, and the next attempt starts with that line in its system prompt.

attempt 1   SELECT sum(total) FROM orders WHERE day = ...   → $41,220
            harness check: expected $52,900 from finance     → FAIL
reflect     "the query ran and returned a number, so this is not a syntax
             error; `orders` is missing rows that `orders_v2` has"
store  +=   "orders_v2 is canonical; orders is a stale view kept for BI."
attempt 2   system prompt = [3 prior lessons + the new one] + task
            SELECT sum(total) FROM orders_v2 WHERE day = ... → $52,900  ✓

Note what did not happen: attempt 2 was not handed the failing query to patch. It was handed the same task plus a fact, and re-derived the query from scratch. That is why the lesson also fixes every later query against that table, where patching the SQL would have fixed exactly one.

Enforce specificity in the reflection prompt, cap the store, and let it evict. In the cross-task version, every lesson is a permanent context tax on every future run.

Cost. 3–10× for the retry loop, plus a standing per-run cost that is easy to underestimate. Price it out:

40 lessons × 25 tokens          = 1,000-token block
resent on all 20 turns of a run = 20,000 input tokens
20,000 · $5/MTok  (Opus)        = $0.10 per run
same, in a cached prefix (0.1×) = $0.01 per run

10,000 runs/day, uncached       = $1,000/day
10,000 runs/day, cached         =   $100/day

That is a $900/day gap, for text nobody reads. It is the entire argument for keeping the store in a stable, cacheable position and capping its size — and for deleting lessons that stop earning, which nothing will do for you.

9. Autonomous Loop

One pattern in this chapter has no human inside it: the agent writes its own to-do list, works through it, and rewrites it as it learns. That arrangement has a failure all its own — drift, where every individual step is defensible and the chain of them is not — and three conditions that make it survivable at all.

The shape, in plain words. Give the loop a goal and a budget. It generates its own list of tasks, executes them one at a time, checks each result, and writes what it learned back into the list — appending, reordering and deleting tasks — until the goal is met, the budget runs out, or a drift check halts it.

The problem it solves. A goal too open-ended to enumerate up front, where the second task genuinely cannot be written until the first one has reported. The running example here is “reduce our p95 latency”. p95 latency is the response time that 95% of requests come in under — the standard way to talk about the slow tail rather than the average.

The cost of solving it that way is that nothing in the loop’s own logic makes it stop.

The diagram is longer than the others because it has three exits, not one. Find them first: Done, Halt + partial report, and Halt + alert. Then look at the two diamonds in the middle, Machine-checkable verification and Drift vs ORIGINAL goal — those are the checks that make the rest survivable.

flowchart TD
    G([Goal + budget]) --> P[Generate tasks]
    P --> Q[[Task queue]]
    Q --> EX[Execute next]
    EX --> VER{Machine-checkable<br/>verification}
    VER --> CR[Self-critique]
    CR --> U[Update queue]
    U --> DR{Drift vs<br/>ORIGINAL goal}
    DR -->|drifted| D([Halt + alert])
    DR -->|ok| ST{Stop?}
    ST -->|goal met| F([Done])
    ST -->|budget out| B([Halt + partial report])
    ST -->|no| Q

    style G fill:#1d3557,color:#fff
    style F fill:#2d6a4f,color:#fff
    style B fill:#bc6c25,color:#fff
    style D fill:#9d0208,color:#fff

Mechanism. The loop starts from Goal + budget. The budget is an input, not a setting, because it is the only thing that makes termination guaranteed.

Generate tasks takes the goal — and, on later laps, what the last execution revealed — and emits the initial Task queue. For “reduce p95 latency” that might be profile endpoints, find slow queries, check cache hit rate.

Execute next pops one item and runs it, usually as a small ReAct or Plan-and-Execute sub-run. That sub-run is where the cost multiplier comes from.

Then two checks, and the reason there are two is the most useful thing in this diagram.

Machine-checkable verification asks did this work, using something that needs no LLM: p95 measured before and after, the test suite exit code, the row count. It is ground truth — cheap and incorruptible — but it can only score the task that was actually run.

Self-critique asks the question no assertion can: was that the right task at all, and what does the result imply about the next one? It is judgement, it is a model call, and it is fallible.

You need both, because verification can pass on a task that should never have been queued. The dashboard got built, the tests are green, and the p95 is unchanged.

Update queue then writes the consequence back: append what execution revealed, reprioritise what the result made urgent, drop items the result made pointless. That write-back is the whole pattern, and it is also the vulnerability.

Against Plan-and-Execute, one sentence. Plan-and-Execute fixes its queue at plan time and only rewrites it on failure; here the agent appends to its own queue after every step.

That single change — a queue that grows from its own output — is what makes drift possible, and it is why this pattern needs a goal-comparison check that Plan-and-Execute does not.

Failure mode — drift. This is the failure characteristic of the pattern, and it is the reason the two extra checks exist. Each self-generated task looks locally reasonable; twenty steps later the agent is optimizing something nobody asked for.

Read the six tasks below in order and watch the subject change from latency to dashboards, one small step at a time:

goal: reduce p95 latency
  → profile endpoints        ✓
  → add caching              ✓
  → refactor the cache layer  ~
  → add cache metrics         ~
  → build a metrics dashboard ✗
  → add dashboard auth        ✗✗

Every arrow is defensible on its own. The chain is not.

Detection: the Drift vs ORIGINAL goal diamond periodically re-scores the queue against the immutable original goal text — never the latest restatement, because restating is exactly how drift launders itself.

Score that exact trace, because the mechanism is only convincing with numbers on it. Ask a cheap model, per queued task: “On a 0–1 scale, how directly does completing this task reduce p95 latency? Quote the clause of the goal it serves.”

Scored against the frozen string reduce p95 latency, the same six tasks come out like this:

profile endpoints          1.0   "reduce p95" — measures the target directly
add caching                0.9   plausible causal path to p95
refactor the cache layer   0.5   maintains a thing that helps; no p95 delta
add cache metrics          0.4   observes the helper, doesn't move p95
build a metrics dashboard  0.1   ← first task that cannot quote the goal
add dashboard auth         0.0   serves the dashboard, not the goal

The first task that cannot quote a clause of the goal is the breach. Here that is build a metrics dashboard — step 5, not step 6 where the wrongness becomes obvious to a human.

Trip at a threshold and take the drifted edge to Halt + alert. Two thresholds that work: halt if the top queued item scores below 0.3, or if the queue’s mean score drops for two consecutive laps.

Note what would have happened had you re-scored against the agent’s current framing — by now something like “improve latency observability”. Every one of these six tasks scores above 0.8 against that. That is the laundering, and freezing the goal string is the only defence.

Three halts, and they are not interchangeable.

Done on goal met is success.

Halt + partial report on budget out is a normal exit. The loop is designed to hit it, so the report has to be genuinely usable: what was completed, what is half-done, what the queue still holds.

Halt + alert on drifted is the only one that pages a human, because it means the loop’s own judgement is no longer trustworthy and no further self-critique will catch that.

Cost. 20–100×+, and unbounded by construction: Update queue can always add one more item, so nothing in the loop’s own logic terminates it.

A hard dollar ceiling is not a safety feature bolted on. It is part of the architecture, and it is what turns Halt + partial report from an error path into a designed outcome.

Make it concrete before you ship. At $0.15 per sub-run and a $30 ceiling, $30 / $0.15 = 200 laps. If the goal plausibly needs more than 200, the budget is wrong — and you will find that out by reading a partial report.

Interview framing: say up front that fully autonomous is usually wrong, name the three conditions that make it survivable — reversible actions, a machine-checkable success signal, a hard budget — then design it anyway. Flagging the risk and delivering beats refusing. Case study 05 is the full build.

Side by side

All nine patterns fit in one table, but two of its columns will mislead you if read naively: “cost” is a token multiplier rather than a dollar figure, and “debuggable” is a claim about replay, not about logging.

Cost is the multiplier defined at the top of the chapter: total input plus output tokens at a fixed model tier, relative to answering the same task in one plain call — and not dollars.

Tiering is a separate lever that moves dollars underneath a fixed multiplier. That is why Routing reads above 1× here and below 1× in dollars (Routing: 0.53).

Routing’s ~1.09× is the token definition applied literally. The 200-in / 5-out classification adds 205 tokens to the 2,400 tokens the handler call (2,000 in / 400 out) was going to read and write regardless:

(205 + 2,400) / 2,400 = 1.085

The familiar “the router is a 1% rounding error” is a different ratio: the dollar one, $0.000225 / $0.0200, taken across two different rungs of the price ladder. It is true and useful, and it is not the number this column reports.

Debuggable is a new column. It counts how many model-driven decisions sit between input and output, and whether you can replay a failure deterministically.

Five stars means every branch was taken by your code, so a failing case reproduces exactly. One star means the model chose the path, the tasks, and when to stop — so the same input may not even fail the same way twice.

Read the “context growth” column first. Only ReAct is quadratic, and it is also the default choice most people reach for. That mismatch is where most agent cost surprises come from.

Then read Cost and Debuggable together — but read them at the ends, not row by row. The cheapest rows are the most debuggable: Prompt Chaining and Routing at five stars. The most expensive row is the least debuggable: Autonomous at 20–100× and one star.

That much is not a coincidence. Every star you lose is a decision you handed to the model, and every decision handed to the model is a call you now pay for and cannot replay.

The middle of the table is not monotonic, though, and the exceptions are informative. Twice, the cost falls as you read down while the stars go up.

First: Orchestrator–Worker at 4–15× and three stars is followed by Evaluator–Optimizer at 2N× and four stars. At the round cap this chapter recommends — N ≤ 3, so 2N ≤ 6× — Evaluator–Optimizer is cheaper as well as more debuggable.

Second: ReAct at 5–20× and two stars is followed by Plan-and-Execute at 3–8× and four stars. This one is worth naming out loud in an interview. Plan-and-Execute costs less and debugs better than ReAct because it makes all of its decisions in one place — a single planner call producing an inspectable queue — rather than one decision per turn scattered through a growing conversation.

The lesson from both exceptions: debuggability tracks where the model decides, not only how much you pay.

What the table has no row for is cheap, flexible, and debuggable all at once.

Real systems compose

No production system is one pattern. The support agent below wires four of them together — a router in front, two different lanes behind it, one shared quality gate, and a single exit to a human. Trace any path from Message to an exit and you cross a router, then one handler, then a gate:

flowchart LR
    M([Message]) --> R{Router}
    R -->|FAQ| RAG[Single RAG call]
    R -->|account| RE[ReAct + tools]
    R -->|angry| H([Human])
    RE --> EV{Grounding gate}
    RAG --> EV
    EV -->|pass| O([Reply])
    EV -->|fail| H

    style R fill:#bc6c25,color:#fff
    style EV fill:#2d6a4f,color:#fff
    style O fill:#2d6a4f,color:#fff

Read that diagram as a chain of four patterns: a router in front, then either a single retrieval-augmented (RAG) call or a ReAct loop, then an evaluator acting as a gate, and finally an escalation to a human.

Naming the composition is the senior answer; naming a single pattern is the junior one. But name it precisely — every box here was chosen against a specific alternative.

Why each branch is what it is

The router’s three classes buy three different cost and risk profiles, not three prompts.

FAQ — a frequently-asked question, the kind a published policy already answers — goes to a single RAG call: one retrieval, one generation, no loop. A policy question has a document that answers it, and an agent loop would only add ways to be wrong.

account goes to ReAct + tools, because “why was I charged twice” cannot be answered without looking up this user’s actual charges, and step 3 depends on what step 2 returns.

angry exits to a human immediately. No model composes a reply, so the branch where a bad answer would do the most damage is also the branch with the least automation.

The grounding gate, and its fail edge

The grounding gate is the Evaluator–Optimizer of Evaluatoroptimizer reduced to a single round: it judges once and never asks for a revision.

It checks exactly one thing. Is every factual claim in the draft reply supported by a document that was actually retrieved, or by a tool result actually returned in this session? Not “is this a good answer” — that is the vague criterion Evaluatoroptimizer warns about.

Every claim must cite a span from the retrieved chunk or a field from the tool response, and an uncited claim fails. That makes the check mechanical on the tool branch — no LLM judge needed — and near-mechanical on the RAG branch.

Both lanes pass through it, and that is deliberate, because each lane hallucinates in its own way. The RAG lane embellishes beyond the passage it retrieved. The ReAct lane narrates a tool result it never actually got. One gate catches both.

The fail edge is the part most designs omit, and the part that makes the composition honest. A grounding failure does not retry, and it does not ship with a hedge. It goes to the human queue — the same terminal as the abusive messages.

That is the whole safety argument in one arrow: the system’s answer to “I cannot support this claim” is a person, not another sample.

It also tells you what to monitor: the escalation rate by branch. Rising on the FAQ lane means retrieval is missing documents. Rising on the account lane means a tool is failing quietly.

Next: 03 — Tools & MCP.

Also from this chapter: 02a — Building One: Evaluator–Optimizer, Three Tiers writes the Evaluatoroptimizer loop three times over — whiteboard pseudocode, LangGraph, and the Anthropic SDK — which is the one pattern here you are most likely to be asked to code on the spot. It is split out because it is a different genre from this file: a code tutorial rather than a comparative catalogue, and the nine patterns above have to stay adjacent for the decision tree and the summary table to work. ReAct gets the same treatment in chapter 01.