This chapter has you write one loop three times, at three levels of fidelity:
- Pseudocode — the version you would put on a whiteboard.
- A graph framework — the same loop expressed as nodes and edges.
- The vendor’s SDK — a production version. SDK stands for software development kit: the official client library for calling the model.
All three are the same loop wearing progressively more clothes. All three also make the same number of model calls — six, at a cap of three rounds — and an implementation that makes seven has a bug rather than a preference, for a reason Tier 1 traces call by call.
Finish it and you can write any of the three from memory, say which parts are framework and which parts are the idea, and defend the two design decisions an interviewer always probes: when the loop stops, and what happens when the judge is wrong.
What the pattern is
An Evaluator–Optimizer is a draft-and-critique loop. It uses three moves:
- One large language model (LLM) call generates an answer — the draft.
- A second call judges that draft against criteria you wrote down in advance — the verdict.
- On a failing verdict, the generator revises, using the judge’s specific complaints.
Repeat until the judge passes the draft, or until a round cap — a hard limit on iterations that you set — stops the loop.
Why the judge is a separate call
The load-bearing detail is that the judge runs as a separate call with a fresh context.
“Fresh context” means the judge’s request contains only the task and the draft. It does not contain the conversation that produced the draft — no earlier attempts, no reasoning, no “here is my answer” framing. As far as the judge can tell, some stranger wrote this text.
That matters because the model reads the draft as input to judge rather than as something I just wrote. A model asked to critique its own visible output in the same conversation tends to defend it. The same model handed the same text cold will find the missing citation.
The full treatment — why the fresh context matters, when to reach for the pattern, how it oscillates, and what it costs — is Evaluatoroptimizer. This file is the code.
Input and output
The loop takes two things in:
- A task in ordinary text — “write the API integration guide for the payments endpoint”.
- A written list of criteria — the standards the judge grades against.
It returns two things:
- The last draft it reached.
- A boolean saying whether that draft actually passed.
Both halves of the return value matter. A loop that hands back a draft without saying whether it passed is indistinguishable from one that succeeded. That is the single most dangerous bug in the pattern: the caller ships a failing draft believing it was approved.
And the draft is the last one, not the best one. Nothing in the loop ranks two drafts against each other. Nothing keeps the highest-scoring one. The judge returns a pass/fail verdict, not a score you could sort on — so there is nothing to sort by even if you wanted to.
In an interview, say those two things in that order. “The last draft, and a flag” first. “The loop keeps no ranking, so it cannot promise the best one” second — because that is the follow-up question.
Only one other pattern in the catalogue gets asked for in code this often, and that is ReAct — built at the same three tiers in chapter 01. Between the two files you have both of the loops you are realistically asked to write on a whiteboard.
Tier 1 — Pseudocode
Start with the shape you would draw on a whiteboard. Read the block below for its three return statements — they are the whole point, and two of the three admit the draft did not pass.
draft = generate(task)
for round in 1..MAX:
verdict = evaluate(task, draft) # fresh context, structured output
if verdict.passed: return draft, True
if round == MAX: return draft, False # judged, and it failed: report the miss
draft = generate(task, draft, verdict.issues)There is no fourth way out of that loop. And every draft that can be returned has been through evaluate first — which is what lets the last return say False honestly, rather than hand back something no evaluator ever looked at.
Counting the calls
The call count is the thing you are most likely to be asked to price on the spot, so trace it rather than reciting it.
One generate runs before the loop. Then each round runs one evaluate, and each round except the last runs one generate to revise. Trace it at MAX = 3:
generate -> d0 (before the loop)
round 1: evaluate d0 -> fail -> generate -> d1
round 2: evaluate d1 -> fail -> generate -> d2
round 3: evaluate d2 -> fail -> return (d2, False)Count the columns: 3 generates (d0, d1, d2) and 3 evaluates. So N generates plus N evaluates — 2N calls at a cap of N rounds, which is six at N = 3.
Why chapter 02 says 2N+1
Evaluatoroptimizer prices the same pattern at 2N+1, seven at N = 3. Both numbers are right, about different loops.
2N+1 counts the version that revises on its way out the door — one that runs generate after the round-3 verdict and returns that new draft. Trace that variant and the last two steps become:
round 3: evaluate d2 -> fail -> generate -> d3 -> return (d3, False)Seven calls, and the extra call is exactly the draft nobody judged. d3 was produced after the last evaluation, so no evaluator ever saw it; the caller receives an unreviewed artifact.
That gives you a rule worth saying out loud: generates and evaluates pair up when you judge everything you might return, so an odd call count is the tell that one draft went out unread.
The verdict schema, which both tiers below need
Before either real implementation, define what a verdict is. Both tiers depend on it, and the reason to write it down first is that it is the only part of this design that is neither framework nor plumbing — it is the contract.
pydantic is a library that turns an ordinary Python class into a machine-checkable description of the data you want back. You declare the fields and their types; pydantic then does two jobs with that one declaration. It produces the schema you send to the model, and it validates the model’s reply into a real Python object.
The class below plays exactly that dual role: it is the schema handed to the judge, and the type the judge’s answer arrives as. Read it for the four fields and the two helper functions — each is unpacked immediately after.
from pydantic import BaseModel, Field
class Verdict(BaseModel):
reasoning: str = Field(description="Cite specific evidence before judging.")
criteria_met: dict[str, bool]
passed: bool
issues: list[str]
# The five names the judge is asked about. Tier 3's CRITERIA block spells the
# same five out in prose; add a line there and add its key here, or `accepted`
# starts approving a criterion nobody checked.
CRITERION_KEYS = ("api_cited", "code_runs", "errors_named",
"sentence_len", "no_invention")
def accepted(v: Verdict) -> bool:
"""`passed` is the model's claim; `criteria_met` is its evidence.
Requiring both — and requiring the evidence to cover every criterion you
asked about — is what stops a verdict of `{"criteria_met": {}, "passed":
true}` from ending the loop with a draft nobody checked.
"""
return (v.passed
and all(k in v.criteria_met for k in CRITERION_KEYS)
and all(v.criteria_met.values()))
def as_feedback(issues: list[str]) -> str:
"""`issues` is a list; a prompt is text. Render it once, here."""
return "\n".join(f"- {i}" for i in issues)The four fields, and why each is there
Field order is not cosmetic here, so start with that.
reasoning comes first because of how constrained decoding works. Constrained decoding is the mechanism that forces the model’s output to match the declared shape, by blocking any token that would break it (Structured output is a guarantee not a request). It generates fields in the order they are declared. Put passed first and you force the model to commit to a verdict before it has written down a single piece of evidence for one. Put reasoning first and the evidence exists before the verdict does.
criteria_met is a dict from criterion name to pass/fail. It reports each named criterion separately, so a failure says which criterion failed rather than just that something did.
passed is the single boolean the loop branches on.
issues is the list fed back to the generator. It exists because a generator told only “rejected” has nothing to change — it will resample a fresh draft rather than revise the one it has.
The two helpers, and why they ship with the schema
Both helpers are places the contract leaks if you skip them.
Gate on accepted, not on v.passed. Nothing in the schema ties the boolean to the dict — they are two independently generated fields. So this is perfectly valid output:
{"reasoning": "", "criteria_met": {}, "passed": true, "issues": []}Schema-valid, and it ends the loop on round 1 against an empty criteria dict. accepted is the one line that makes criteria_met load-bearing rather than decorative: it demands the claim (passed), demands the evidence covers all five keys, and demands every one of those five is True.
Render with as_feedback, not the raw list. issues is a list[str], and what goes into the next prompt is text. Interpolate the list directly into an f-string and the generator is shown this:
['add caveats', 'cite section']That is a stringified Python list — brackets, quotes, and commas included — where you meant two bullet points. It is one line to fix, and it is the line both tiers below would otherwise get wrong in the same way.
Tier 2 — LangGraph
LangGraph is a library that expresses an agent as a graph. Nodes are ordinary functions. Edges are the allowed transitions between them. The framework runs the graph, carrying a shared state dictionary from node to node: each node reads the state, returns a partial update, and the framework merges that update in.
The state’s keys and their types are declared up front with a TypedDict — a Python class that describes the shape of a dictionary (which keys it has, and what type each value is) without changing the fact that it is a plain dict at runtime. The S class in the listing below is exactly that: the loop’s state, spelled out.
Verdict, accepted, and as_feedback are the ones from the section above, reused rather than repeated. That reuse is real, not cosmetic — what runs below is that section’s copy of the two helpers, so mutating them is not a no-op.
The stand-in that makes the listing runnable
Neither langgraph nor langchain-anthropic is a dependency of this repo, and a listing that nothing executes is a listing that quietly rots. So the block below registers a stand-in — a fake module that satisfies the imports — under both names, unconditionally. Every machine then runs the same thing, rather than something that depends on what happens to be installed.
The stand-in models exactly four graph semantics, and nothing else:
- A node is a function from state to a partial state dict.
- The runner merges each returned dict into the state.
- The router returns the name of the next node.
- A key the schema does not declare never becomes part of the state.
Checkpointing, resume, and streaming — the reasons point 1 below gives for reaching for a graph framework at all — are simply absent. That is the deal, and it bounds what the next section can claim: what gets executed here is this chapter’s node and routing logic, not the framework.
Skip this block on a first read. It is scaffolding. The pattern is the listing after it.
# A stand-in for the two imports in the listing below. It is a model of four
# graph semantics, NOT LangGraph, and it never calls a model.
import sys
import types
START, END = "__start__", "__end__"
CALLS: list = [] # "generate" / "evaluate", in the order the graph makes them
PROMPTS: list = [] # the prompt text each of those calls was handed
INITIAL: list = [] # the state dict every invoke() was seeded with
def canned(passed: bool, issues=("add caveats", "cite section"), met=None) -> Verdict:
"""One scripted verdict. `met=None` means 'evidence agrees with the claim'."""
return Verdict(reasoning="cited section 4.2", passed=passed, issues=list(issues),
criteria_met=dict.fromkeys(CRITERION_KEYS, passed) if met is None
else met)
# Verdicts the judge returns, oldest first. Four of them, and the demo run at
# the end of the listing below should need exactly one: a loop that runs on
# past a passing verdict has to be caught by a named assertion in the next
# section, not by this stub running out of replies.
SCRIPT = [canned(True, issues=[])] * 4
class MiniChat:
"""Records the prompt, returns the next scripted reply. No network."""
def __init__(self, **kw):
self.schema = None # set by with_structured_output
def with_structured_output(self, schema):
self.schema = schema
return self
def invoke(self, prompt):
PROMPTS.append(prompt)
if self.schema is None: # the generator: replies are text
CALLS.append("generate")
return types.SimpleNamespace(content=f"d{CALLS.count('generate') - 1}")
CALLS.append("evaluate") # the judge: replies are Verdicts
if not SCRIPT:
raise AssertionError("the judge ran more rounds than the script allows")
return SCRIPT.pop(0)
class MiniStateGraph:
"""Nodes, one conditional branch, and a merge. That is the whole model."""
def __init__(self, schema):
self.schema, self.nodes, self.edges, self.branch = schema, {}, {}, {}
def add_node(self, name, fn):
self.nodes[name] = fn
def add_edge(self, src, dst):
self.edges[src] = dst
def add_conditional_edges(self, src, router, allowed):
self.branch[src] = (router, list(allowed))
def compile(self):
return self
def invoke(self, state, max_steps=64):
state = dict(state) # NOTHING is defaulted: a missing key is a KeyError
INITIAL.append(dict(state)) # recorded so the next section can read the seed
node = self.edges[START]
for _ in range(max_steps): # a graph that never reaches END is an error here,
if node == END: # not a hang
return state
fn = self.nodes[node]
for k, v in (fn(state) or {}).items():
if k in self.schema.__annotations__: # a key the schema does not
state[k] = v # declare is dropped, not merged
if node in self.branch:
router, allowed = self.branch[node]
node = router(state)
assert node in allowed, node
else:
node = self.edges[node]
raise AssertionError("the graph never reached END")
sys.modules["langgraph"] = types.ModuleType("langgraph")
sys.modules["langgraph.graph"] = types.ModuleType("langgraph.graph")
sys.modules["langgraph.graph"].StateGraph = MiniStateGraph
sys.modules["langgraph.graph"].START = START
sys.modules["langgraph.graph"].END = END
sys.modules["langchain_anthropic"] = types.ModuleType("langchain_anthropic")
sys.modules["langchain_anthropic"].ChatAnthropic = MiniChatThe listing
This is the pattern. Read it top to bottom in four beats: the two model clients, the state class S, the three functions (generate, evaluate, route), and then the graph wiring that connects them.
from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langchain_anthropic import ChatAnthropic
# max_tokens is a truncation ceiling, not a budget: a reply that reaches it
# comes back cut off mid-sentence with `stop_reason: "max_tokens"`. 8192 for a
# draft, 2048 for a verdict that is four short fields.
llm = ChatAnthropic(model="claude-opus-5", max_tokens=8192)
# with_structured_output is what makes v.issues / v.passed exist — without it,
# judge.invoke returns text and you are back to parsing prose.
judge = ChatAnthropic(model="claude-opus-5", max_tokens=2048).with_structured_output(Verdict)
class S(TypedDict):
task: str
draft: str
feedback: str # text, not v.issues — see as_feedback above
rounds: int
passed: bool # written by evaluate, read by route — must be declared
def generate(s: S) -> S:
prompt = s["task"] if not s["draft"] else (
f"{s['task']}\n\nPrevious:\n{s['draft']}\n\nFix:\n{s['feedback']}")
return {"draft": llm.invoke(prompt).content, "rounds": s["rounds"] + 1}
def evaluate(s: S) -> S:
v = judge.invoke(f"Task: {s['task']}\nDraft: {s['draft']}")
return {"feedback": as_feedback(v.issues), "passed": accepted(v)}
def route(s: S) -> str:
return END if s.get("passed") or s["rounds"] >= 3 else "generate"
g = StateGraph(S)
g.add_node("generate", generate)
g.add_node("evaluate", evaluate)
g.add_edge(START, "generate")
g.add_edge("generate", "evaluate")
g.add_conditional_edges("evaluate", route, ["generate", END])
app = g.compile()
# Seed EVERY key in S. LangGraph does not default a missing one, so
# `app.invoke({"task": "..."})` raises KeyError: 'draft' inside the first node,
# and adding only `draft` then raises KeyError: 'rounds'.
final = app.invoke({"task": "write the payments integration guide",
"draft": "", "feedback": "", "rounds": 0, "passed": False})
print(final["passed"], final["draft"][:80])Four things to say about this block, because the framework-specific idea is easy to miss:
routeis a conditional edge, not awhile. That is the entire reason to reach for LangGraph here. The loop lives in the graph, so the retry edgeevaluate → generateis a declared part of the structure rather than a line of imperative code — which means the framework can checkpoint state between nodes, resume a run that died on round 2, and stream node-by-node progress. Awhileloop inoptimize()gets you none of that, which is why Tier 3 has to own its own state.passedmust be inS. Nodes return partial dicts that get merged into the state, soevaluatewriting{"passed": ...}androutereadings.get("passed")only work if the key is declared — a key missing from theTypedDictis a type error, and at runtimeroutesilently readsNoneand loops until the round cap.routechecksrounds >= 3in the same expression aspassed. The cap is part of the edge condition, not a separate guard, so there is no path through the graph that iterates without it. This is the oscillation fix from Evaluatoroptimizer expressed as graph structure rather than as discipline.- The cap counts generates here and evaluates in Tier 3 — and lands on the same number.
roundsis incremented bygenerate, androuteruns afterevaluate, so a cap of 3 buys three generates and three evaluates: six calls, and the last draft is judged beforeroutesees it. Tier 3 counts rounds of judging instead and stops at the same six. Same loop, different fencepost, same bill — and in both, the draft that comes out the end has a verdict attached to it.
That last point is a fencepost worth walking rather than trusting. Follow one pass through the graph:
generate rounds 0 -> 1, writes d0 route not consulted (unconditional edge)
evaluate d0 -> fail route: passed False, rounds 1 < 3 -> generate
generate rounds 1 -> 2, writes d1
evaluate d1 -> fail route: passed False, rounds 2 < 3 -> generate
generate rounds 2 -> 3, writes d2
evaluate d2 -> fail route: rounds 3 >= 3 -> END, returns d2Three generates, three evaluates, and the returned draft d2 was evaluated on the step immediately before the graph ended. Same six calls as Tier 3, reached by counting a different thing.
Proving Tier 2’s logic holds
Those four points are claims about what the code does, and a claim about code is worth exactly as much as the thing that recreates it failing. The block below drives the graph above against the stand-in: it exercises the two node functions and the router, and nothing else. LangGraph is not under test here — the runtime underneath is the model registered before the listing, so read every assertion below as the nodes and the routing are right, never as the framework works.
Each assertion names the defect it pins, and every one was checked by putting that defect back and watching this block go red. The single exception is a seed cut down to task alone: that never reaches an assertion at all, because the listing above dies in its first node — which is the failure the try in assertion 2 spells out, and the reason the honest place to catch a partial seed is the KeyError itself.
Before reading the assertions, know what the three recorders hold. CALLS is the ordered list of "generate" / "evaluate" strings, one per model call the graph made. PROMPTS is the text each of those calls was handed. INITIAL holds the state dict each invoke() started from — so INITIAL[0] is the exact seed the listing above ran with, which is what SEED reuses.
SEED = dict(INITIAL[0]) # the exact dict the listing above invoked with
# ---- 1. `passed` is declared in S, so evaluate's verdict reaches route.
# Undeclared, the update is dropped on the merge, `route` reads None,
# and a draft that passed on round 1 gets revised twice more anyway.
CALLS.clear(); PROMPTS.clear(); SCRIPT[:] = [canned(True, issues=[])] * 4
final = app.invoke(SEED) # 4 verdicts scripted, 1 legitimately used:
# a loop that ignores `passed` runs on, and
# is caught by an assertion, not by the stub
assert final["passed"] is True, "evaluate wrote `passed` and route never saw it"
assert CALLS == ["generate", "evaluate"], ("a passing verdict did not stop the "
"graph", CALLS)
assert final["draft"] == "d0", final["draft"]
# ---- 2. every key in S is seeded before invoke. A partial initial state dies
# in the first node, before the first model call.
assert set(SEED) == set(S.__annotations__), (
"app.invoke seeds a different key set than S declares: "
f"{sorted(set(S.__annotations__) ^ set(SEED))}")
try:
app.invoke({"task": "write the payments integration guide"})
raise AssertionError("a partial initial state was accepted")
except KeyError as e:
assert "draft" in str(e), e
# ---- 3. the cap: three generates, three evaluates, and the draft that comes
# out was the last one judged. 2N, the same six calls as Tier 3.
CALLS.clear(); PROMPTS.clear(); SCRIPT[:] = [canned(False)] * 8
final = app.invoke(SEED)
assert CALLS == ["generate", "evaluate"] * 3, ("the cap is not 3 rounds", CALLS)
assert final["rounds"] == 3, final["rounds"]
assert (final["passed"], final["draft"]) == (False, "d2"), final
assert "d2" in PROMPTS[-1], "the returned draft is not the one last judged"
# ---- 4. feedback reaches the next prompt as TEXT, not as a repr'd list.
CALLS.clear(); PROMPTS.clear(); SCRIPT[:] = [canned(False), canned(True, issues=[])]
final = app.invoke(SEED)
assert isinstance(final["feedback"], str), type(final["feedback"])
assert "- add caveats\n- cite section" in PROMPTS[2], PROMPTS[2]
assert "['add caveats'" not in PROMPTS[2], "the raw issues list reached the generator"
assert "d0" in PROMPTS[2], "the revision prompt does not carry the previous draft"
# ---- 5. the node calls `accepted`, so the claim alone does not end the loop.
CALLS.clear(); SCRIPT[:] = [canned(True, met={}, issues=[])] * 3
final = app.invoke(SEED)
assert final["passed"] is False, "an empty criteria_met ended the loop"
assert CALLS == ["generate", "evaluate"] * 3, CALLS
print("02a Tier 2: node and routing assertions hold")What this deliberately does not claim. Three things, and they are the reason the block above is short:
- The merge rule is modelled, not verified. Point 2 says an undeclared key leaves
routereadingNone; the stand-in drops undeclared keys because that is the behaviour being described, so assertion 1 proves the nodes depend onpassedreaching the state, not that LangGraph drops it. Some versions raise instead — in which case the same edit fails louder, not quieter. - Checkpointing, resume and streaming are untested. They are point 1’s entire argument for using a graph framework, and nothing here touches them. That claim rests on the library’s documentation.
- No model is called, so nothing here exercises truncation, refusal or a reply with no text in it. Tier 3 tests those on the SDK path, where the code that handles them lives; Tier 2 delegates them to
langchain-anthropicand this file has no opinion on how well that goes.
Tier 3 — Anthropic SDK
Now the same loop with no framework at all — just the vendor’s Python client. Verdict, CRITERION_KEYS, accepted, and as_feedback are repeated here, unchanged, so this listing runs on its own.
It is the longest block in the chapter, so read it in five beats:
- The schema and helpers, copied from above.
JUDGE_SYSTEMandCRITERIA— the judge’s instructions and the five criteria in prose.first_text— the reply reader, which is where three failure modes are caught.generateandevaluate— one model call each.optimize— the loop itself, in eleven lines.
from __future__ import annotations # `list[str] | None` needs this on 3.9
import anthropic
from pydantic import BaseModel, Field
client = anthropic.Anthropic()
class Verdict(BaseModel):
reasoning: str = Field(description="Cite specific evidence before judging.")
criteria_met: dict[str, bool]
passed: bool
issues: list[str]
CRITERION_KEYS = ("api_cited", "code_runs", "errors_named",
"sentence_len", "no_invention")
def accepted(v: Verdict) -> bool: # the claim AND the evidence
return (v.passed
and all(k in v.criteria_met for k in CRITERION_KEYS)
and all(v.criteria_met.values()))
def as_feedback(issues: list[str]) -> str:
return "\n".join(f"- {i}" for i in issues)
class NotAnAnswer(RuntimeError):
"""The reply stopped for a reason that is not a finished draft."""
JUDGE_SYSTEM = (
"You are a strict evaluator. Judge the draft against every criterion. "
"Report each unmet criterion as one concrete, actionable issue. "
"Do not rewrite the draft."
)
# This is what "clear, articulable criteria" actually looks like. Each line is
# independently checkable and independently reportable — that is what makes
# `criteria_met: dict[str, bool]` meaningful instead of decorative. The five
# names here are CRITERION_KEYS, and the two lists have to move together.
CRITERIA = """<criteria>
api_cited: Every claim about API behaviour cites a section number from the spec.
code_runs: Every code sample is valid Python 3.9 and imports only the stdlib.
errors_named: The error-handling section names all four documented failure codes.
sentence_len: No sentence exceeds 40 words.
no_invention: No feature is described that does not appear in the spec.
</criteria>"""
def first_text(r) -> str:
"""The only safe way to read a reply.
Never `r.content[0]` and never a bare `next(...)` over it: a refusal
arrives as an ordinary HTTP 200 whose content list is EMPTY, so both
spellings crash the loop instead of reporting a decline.
"""
if r.stop_reason == "refusal":
raise NotAnAnswer(f"model declined: {getattr(r, 'stop_details', None)}")
if r.stop_reason == "max_tokens":
raise NotAnAnswer("truncated at max_tokens: raise the ceiling or split the task")
text = next((b.text for b in r.content if b.type == "text"), None)
if text is None: # thinking blocks only, or nothing at all
raise NotAnAnswer(f"no text block in the reply ({r.stop_reason})")
return text
def generate(task: str, draft: str = "", issues: list[str] | None = None) -> str:
prompt = task if not draft else (
f"{task}\n\n<previous_draft>\n{draft}\n</previous_draft>\n"
f"<issues>\n{as_feedback(issues or [])}\n</issues>\n"
"Produce a revised version that resolves every issue."
)
r = client.messages.create(
model="claude-opus-5",
max_tokens=8192, # truncation ceiling, not a budget
messages=[{"role": "user", "content": prompt}],
)
return first_text(r)
def evaluate(task: str, draft: str) -> Verdict:
r = client.messages.parse(
model="claude-opus-5",
max_tokens=2048, # a verdict is four short fields
system=[{ # byte-identical every round
"type": "text",
"text": JUDGE_SYSTEM + "\n\n" + CRITERIA,
"cache_control": {"type": "ephemeral"},
}],
messages=[{"role": "user",
"content": f"<task>{task}</task>\n<draft>{draft}</draft>"}],
output_format=Verdict,
)
return r.parsed_output
def optimize(task: str, max_rounds: int = 3) -> tuple[str, bool]:
draft = generate(task)
for round_no in range(1, max_rounds + 1):
v = evaluate(task, draft) # every draft that can be returned
if accepted(v): # goes through here first
return draft, True
if round_no == max_rounds:
return draft, False # judged, and it failed. Never fake success
draft = generate(task, draft, v.issues)
return draft, False # max_rounds < 1: nothing was judged,
# so this path can only ever be FalseSeven things to narrate while writing this. Each has its own subsection below.
1. reasoning is the first field in the schema
Constrained decoding generates fields in order (Structured output is a guarantee not a request), so putting the score first would force the model to commit before reasoning. Order is not cosmetic.
2. Branch on stop_reason before you touch content
stop_reason is the field on a reply that says why generation stopped. content is the list of blocks the model produced. The rule is: ask why generation stopped before you read what it produced — and never index into content blindly.
first_text exists because three of the endings are not a finished draft, and two of those carry no text block at all:
stop_reason | What actually came back | What the naive read does |
|---|---|---|
refusal | HTTP 200, empty content list | r.content[0].text raises IndexError; next(b.text for b in ...) raises StopIteration |
max_tokens | A draft cut off mid-word | Hands a truncated fragment to the judge as finished work |
end_turn, thinking only | A thinking block, no text block | Same empty-generator crash as the refusal row |
Two rows deserve a note. A refusal is not an HTTP error — it is a normal 200 response whose content list is empty, so the loop dies with a traceback about iteration instead of reporting that the model declined. And on claude-opus-5, adaptive thinking is on by default, so a reply routinely leads with a thinking block; a response truncated before it reaches any text is the same empty-generator crash again.
One helper, three failures.
3. The judge’s system prompt is marked for caching — and as written it will not cache
Start with the mechanism. Prompt caching lets the provider store its precomputed internal state for a block of prompt text, so later requests that begin with the same bytes skip recomputing it. The cache_control marker on the system block is what requests that. Because the block is byte-identical every round, rounds 2 and later would read the criteria at roughly a tenth of the normal input rate (Prompt caching derived).
The mechanism is right. The size is not.
A cached prefix has to clear the model’s minimum, which is 512 tokens on claude-opus-5. Here is the block being cached, measured:
| Quantity | Value |
|---|---|
Characters in JUDGE_SYSTEM + "\n\n" + CRITERIA | 549 |
| Words | 81 |
| Estimated tokens at ~4 characters per token | 549 / 4 ≈ 137 |
Minimum cacheable prefix on claude-opus-5 | 512 |
137 against a floor of 512. The ~4-characters-per-token figure is a rule of thumb (Tokens), but it does not have to be exact — 549 characters is nowhere near 512 tokens under any tokenization, because that would require fewer than 1.1 characters per token.
Below the minimum, nothing caches — and there is no error. The only symptom is cache_creation_input_tokens: 0 in the usage block (Evaluatoroptimizer states the same caveat in as many words).
So: keep the marker. It costs nothing, and it starts paying the moment the criteria list grows to the size that makes the pattern worth using. But verify rather than assume — print usage.cache_read_input_tokens on round 2, and if it is zero you are paying full price for every round.
One more thing to know before you rely on it. "ephemeral" is the short-lived cache tier, and its default time-to-live is five minutes — which three claude-opus-5 calls at high effort can genuinely exceed. It is not “far longer than a revision loop lasts”; it is the same order of magnitude. A one-hour tier exists for exactly that reason, at a higher write price: writing a five-minute entry costs about 1.25× the base input rate, and writing a one-hour entry costs about 2×.
4. max_tokens is a truncation ceiling, not a budget
It is the one parameter here that silently changes behaviour. The model is not told about it, so it does not wrap up as it approaches the limit — it is simply cut off, and you get stop_reason: "max_tokens" with a half-finished draft. 8192 for the generator and 2048 for a four-field verdict are sized to that, and first_text is what stops a truncation being mistaken for a draft.
5. output_format guarantees the verdict parses
Invalid output has probability zero, because the runtime never lets an invalid token be chosen. No regular expressions patching up the text afterwards, and no repair loop. It guarantees shape, though, not intent — which is the next point.
6. accepted, not v.passed
Nothing in the schema ties the boolean to the dict, so {"reasoning": "", "criteria_met": {}, "passed": true, "issues": []} is a schema-valid verdict that ends the loop on round 1 having checked nothing. Gate on the claim and the evidence, and require the evidence to name all five criteria.
7. The failure path returns False, and the draft it returns has been judged
An agent that reports success on round 3 without passing is the most dangerous bug in this pattern. A loop that returns a draft generated after the last verdict is the quieter cousin of it — the call is wasted, the flag is honest, and the artifact is unreviewed. Exiting after the evaluation rather than after the revision fixes both at once.
Proving the guards hold
Six of the seven points above are claims about what the code does when something goes wrong, and a claim about a failure path is worth exactly as much as the test that recreates it. The block below stubs the client out — no network, no credentials — and drives each path.
Each assertion is written so that reverting the guard it pins makes it fail. That is a stricter bar than it sounds: a test that only checked “something was raised” would let all three stop_reason branches be deleted one at a time without ever going red, so each assertion below checks the specific message its branch produces.
The first half of the block is fixtures rather than tests — block, reply, fake, and verdict build canned responses and a client that records traffic into CALLS and JUDGED. The numbered comments mark where the actual assertions start.
import types
CALLS: list = [] # every API call the loop makes, in order
JUDGED: list = [] # the prompt every evaluator actually saw
_real_client = client
def block(kind: str, text: str = ""):
b = types.SimpleNamespace(type=kind)
if kind == "text":
b.text = text # a thinking block has no .text at all
return b
def reply(stop_reason: str, *blocks):
return types.SimpleNamespace(stop_reason=stop_reason, content=list(blocks),
stop_details=None)
def fake(replies, verdicts=()):
"""A client that hands back canned replies and records the traffic."""
rs, vs = list(replies), list(verdicts)
def create(**kw):
CALLS.append("generate")
return rs.pop(0)
def parse(**kw):
CALLS.append("evaluate")
JUDGED.append(kw["messages"][0]["content"])
return types.SimpleNamespace(parsed_output=vs.pop(0))
return types.SimpleNamespace(
messages=types.SimpleNamespace(create=create, parse=parse))
def verdict(passed, met=None, issues=("add caveats", "cite section")):
return Verdict(reasoning="cited section 4.2", passed=passed, issues=list(issues),
criteria_met=dict.fromkeys(CRITERION_KEYS, True) if met is None
else met)
drafts = lambda n: [reply("end_turn", block("text", f"d{i}")) for i in range(n)]
FAILS = [verdict(False, met=dict.fromkeys(CRITERION_KEYS, False))] * 9
# ---- 1. refusal: HTTP 200, EMPTY content. A bare next() raises StopIteration.
client = fake([reply("refusal")])
try:
generate("t")
raise AssertionError("a refusal was read as a draft")
except NotAnAnswer as e:
assert "declined" in str(e), e
# ---- 2. max_tokens: a truncated draft is not a draft.
client = fake([reply("max_tokens", block("text", "half a dra"))])
try:
generate("t")
raise AssertionError("a truncated draft was returned as finished")
except NotAnAnswer as e:
assert "max_tokens" in str(e), e
# ---- 3. thinking-only content: opus-5 thinks by default, so the first block
# is routinely not text, and a truncation before the text is empty too.
client = fake([reply("end_turn", block("thinking"))])
try:
generate("t")
raise AssertionError("a reply with no text block was read as a draft")
except NotAnAnswer as e:
assert "no text block" in str(e), e
# ---- 4. the draft that comes out has been judged, and no call is wasted.
CALLS.clear(); JUDGED.clear()
client = fake(drafts(9), FAILS)
assert optimize("t", max_rounds=3) == ("d2", False)
assert CALLS == ["generate", "evaluate"] * 3, CALLS # 2N, not 2N+1
assert "<draft>d2</draft>" in JUDGED[-1], "returned a draft no evaluator saw"
# ---- 5. `passed` is a claim; `criteria_met` is the evidence.
client = fake(drafts(9), [verdict(True, met={}, issues=[])] * 3) # empty dict
assert optimize("t", max_rounds=3) == ("d2", False), "empty criteria_met passed"
client = fake(drafts(9), [verdict(True, met={"api_cited": True}, issues=[])] * 3)
assert optimize("t", max_rounds=3) == ("d2", False), "one-of-five criteria passed"
client = fake(drafts(9), [verdict(True, met={**dict.fromkeys(CRITERION_KEYS, True),
"no_invention": False}, issues=[])] * 3)
assert optimize("t", max_rounds=3) == ("d2", False), "a failing criterion passed"
client = fake(drafts(1), [verdict(True, issues=[])]) # and a real pass passes,
assert optimize("t", max_rounds=1) == ("d0", True) # or the gate is `return False`
# ---- 6. the step cap halts, at every cap including 0 and negative.
for cap in (0, -1, 1, 3):
CALLS.clear()
client = fake(drafts(9), list(FAILS))
_, ok = optimize("t", max_rounds=cap)
assert ok is False, cap
assert len(CALLS) == max(1, 2 * cap), (cap, CALLS)
# ---- 7. feedback reaches the prompt as text, not as a repr'd list.
assert as_feedback(["add caveats", "cite section"]) == "- add caveats\n- cite section"
assert isinstance(as_feedback([]), str)
client = _real_client
print("02a guards: all assertions hold")Three of those blocks are worth a note.
Assertion 4 pins the call sequence, not just the answer. CALLS == ["generate", "evaluate"] * 3 is what catches the wasted call: the loop that revises on its way out produces a seventh entry and returns d3, a draft whose only reader is the caller. Asserting on the returned tuple alone would let that regress the moment someone “simplified” the exit.
Assertion 5 asserts a pass as well as three failures. Three tests that all expect False are equally satisfied by a gate that reads return False, which is not a gate at all. The fourth case — a genuinely passing verdict that must come back True — is what keeps the other three honest.
Assertion 6 is the good news. The step cap is the one guard in this file that could not be defeated: for round_no in range(1, max_rounds + 1) halts at every cap tried, including 0 and -1, where the loop body never runs at all and the function returns after a single generate with passed=False. That is why the expected call count is max(1, 2 * cap) — at cap 0 or -1 there is exactly one call (the pre-loop generate) and no evaluate at all, and at caps 1 and 3 it is the usual 2N. There is no path through optimize that iterates without the cap, and no cap value that turns the honest False into a True.
The criteria are the design decision
The last thing to say is about the CRITERIA block itself. This is where the advice in Evaluatoroptimizer — if a criterion can be checked in code, check it in code — stops being a slogan and becomes a line-by-line decision.
Two of those five criteria should never reach the model.
code_runs is ast.parse — the standard-library call that parses Python and raises if the syntax is invalid — plus a check of what the sample imports. sentence_len is a regular expression, a pattern matched over the text. Both are a few lines of ordinary code.
Checking them in code buys three things. They become deterministic — same input, same verdict, every time. They become free — no tokens, no latency. And they become immune to a judge having an opinion about whether a 41-word sentence is really that bad.
It also shrinks the judge’s job down to the three criteria that genuinely need reading comprehension: api_cited, errors_named, no_invention.
Why the named keys make this incremental
You do not have to move all of it at once, and the criteria_met dict is what lets you move it one key at a time.
The same dict can be filled by code for some keys and by the model for others. accepted reads the merged dict and cannot tell the difference — which is the point. Concretely, a partially migrated round looks like this:
from code: {"code_runs": True, "sentence_len": False}
from judge: {"api_cited": True, "errors_named": True, "no_invention": True}
merged: all five keys present -> accepted() returns False (sentence_len)This is also why accepted insists every key in CRITERION_KEYS is present. A key that no one filled in is a criterion nobody checked, and silently dropping it would turn a five-criterion gate into a four-criterion one.
Keep the code-checked lines in the prompt anyway. The generator reads the same criteria block, and a criterion the generator never sees is one it will keep violating — you would be catching the violation in code instead of preventing it.
One consistency note
The criteria are about someone else’s code, and this file is code too. The listing above targets Python 3.9 — which is why from __future__ import annotations is there to make list[str] | None legal, and why code_runs asks for valid 3.9 rather than 3.10.
Pick one floor and let every line in the file agree with it. Two floors in one file is how a criterion ends up passing a sample the code cannot import.
What interviewers probe: “How do you know when to stop iterating?” and “What if the evaluator is wrong?” — cap the rounds and prefer code-checkable criteria; calibrate the judge against ~50 human labels — score the same 50 items by hand, compare, and report how often the judge agreed — before trusting it (Calibration what an agreement rate actually means).
Next: 03 — Tools & MCP. The pattern this file builds is catalogued alongside the other eight in chapter 02.