Building an Evaluator-Optimizer Loop

In this lesson, we’ll build one draft-and-critique loop three times, each version at higher fidelity than the last. By the end you’ll be able to write the loop from scratch, count its model calls exactly, and catch the one bug that ships a draft no judge ever read.

Pseudocode.
A graph framework: the same loop as nodes and edges.
The vendor’s SDK: a production version. SDK stands for software development kit: the official client library for calling the model.

All three are the same loop, and all three make the same number of model calls: six, at a cap of three rounds. A version that makes seven has a specific bug, and we trace it below.

Two design decisions settle whether the loop is correct: when it stops, and what happens when the judge is wrong. We’ll keep returning to those two.

What the pattern is

An Evaluator-Optimizer is a draft-and-critique loop with three moves:

One large language model (LLM) call generates an answer, the draft.
A second call judges that draft against criteria written down in advance, the verdict.
On a failing verdict, the generator revises, using the judge’s specific complaints.

Repeat until the judge passes the draft, or until a round cap, a hard limit on iterations that you set, stops the loop.

flowchart TD
    A[Generate draft] --> B[Evaluate draft against criteria]
    B --> C{Passed?}
    C -->|Yes| D[Return draft, True]
    C -->|No| E{Round cap reached?}
    E -->|Yes| F[Return draft, False]
    E -->|No| G[Revise using issues]
    G --> B

Why the judge is a separate call

The judge runs as a separate call with a fresh context. “Fresh context” means the judge’s request contains only the task and the draft, not the conversation that produced the draft, no earlier attempts, no “here is my answer” framing.

That matters because the model then reads the draft as input to judge, not as something I just wrote. A model asked to critique its own visible output in the same conversation tends to defend it. The same model handed the same text cold will find the missing citation.

The design rationale (why the fresh context matters, when to use the pattern, how it oscillates, and what it costs) is covered in the design-patterns chapter. This lesson covers the code.

Input and output

The loop takes two things in:

A task in ordinary text, “write the API integration guide for the payments endpoint”.
A written list of criteria: the standards the judge grades against.

It returns two things:

The last draft it reached.
A boolean saying whether that draft actually passed.

Both halves of the return value matter. A loop that hands back a draft without saying whether it passed is indistinguishable from one that succeeded. This is the most dangerous bug in the pattern: the caller ships a failing draft believing it was approved.

And the draft is the last one, not the best one. Nothing in the loop ranks two drafts against each other or keeps the highest-scoring one. The judge returns a pass/fail verdict, not a score you could sort on, so there is nothing to sort by even if you wanted to.

The other loop worth building at these same three tiers is ReAct, in the agent-foundations chapter.

Tier 1 — Pseudocode

The pseudocode has three return statements; two of them report that the draft did not pass.

draft = generate(task)
for round in 1..MAX:
    verdict = evaluate(task, draft)          # fresh context, structured output
    if verdict.passed: return draft, True
    if round == MAX:   return draft, False   # judged, and it failed: report the miss
    draft = generate(task, draft, verdict.issues)

There is no fourth way out of the loop. Every draft that can be returned has passed through evaluate first, which is what lets the final return report False honestly instead of handing back a draft no evaluator saw.

Counting the calls

Trace the count instead of memorizing it. One generate runs before the loop. Then each round runs one evaluate, and each round except the last runs one generate to revise. At MAX = 3:

        generate  ->  d0                      (before the loop)
round 1: evaluate d0 -> fail  ->  generate -> d1
round 2: evaluate d1 -> fail  ->  generate -> d2
round 3: evaluate d2 -> fail  ->  return (d2, False)

That is 3 generates (d0, d1, d2) and 3 evaluates: N generates plus N evaluates, so 2N calls at a cap of N rounds, which is six at N = 3.

Why the design chapter says `2N+1`

The design-patterns chapter prices the same pattern at 2N+1, seven at N = 3. Both numbers are right, about different loops. 2N+1 counts the version that revises on its way out the door, one that runs generate after the round-3 verdict and returns that new draft:

round 3: evaluate d2 -> fail  ->  generate -> d3  ->  return (d3, False)

Seven calls, and the extra call is exactly the draft nobody judged. d3 was produced after the last evaluation, so no evaluator ever saw it; the caller receives an unreviewed artifact. The rule: generates and evaluates pair up when you judge everything you might return, so an odd call count means one draft went out unread.

The verdict schema, which both tiers below need

Before either real implementation, define what a verdict is. Both tiers depend on it, and it comes first because it is the only part of this design that is neither framework nor plumbing. It is the contract.

pydantic is a library that turns an ordinary Python class into a machine-checkable description of the data you want back. You declare the fields and their types; pydantic then produces the schema you send to the model and validates the model’s reply into a real Python object. The class below plays both roles at once.

from pydantic import BaseModel, Field


class Verdict(BaseModel):
    reasoning: str = Field(description="Cite specific evidence before judging.")
    criteria_met: dict[str, bool]
    passed: bool
    issues: list[str]


# The five names the judge is asked about. Tier 3's CRITERIA block spells the
# same five out in prose; add a line there and add its key here, or `accepted`
# starts approving a criterion nobody checked.
CRITERION_KEYS = ("api_cited", "code_runs", "errors_named",
                  "sentence_len", "no_invention")


def accepted(v: Verdict) -> bool:
    """`passed` is the model's claim; `criteria_met` is its evidence.

    Requiring both — and requiring the evidence to cover every criterion you
    asked about — is what stops a verdict of `{"criteria_met": {}, "passed":
    true}` from ending the loop with a draft nobody checked.
    """
    return (v.passed
            and all(k in v.criteria_met for k in CRITERION_KEYS)
            and all(v.criteria_met.values()))


def as_feedback(issues: list[str]) -> str:
    """`issues` is a list; a prompt is text. Render it once, here."""
    return "\n".join(f"- {i}" for i in issues)

The four fields

Field order matters here.

reasoning comes first because of how constrained decoding works. Constrained decoding is the mechanism that forces the model’s output to match the declared shape, by blocking any token that would break it (structured output is a guarantee, not a request). It generates fields in the order they are declared. Put passed first and the model commits to a verdict before writing down any evidence for one; put reasoning first and the evidence exists before the verdict does.

criteria_met is a dict from criterion name to pass/fail. It reports each criterion separately, so a failure says which criterion failed, not just that something did.

passed is the single boolean the loop branches on.

issues is the list fed back to the generator. A generator told only “rejected” has nothing to change, so it resamples a fresh draft instead of revising the one it has.

The two helpers

Both helpers plug places the contract leaks if you skip them.

Gate on accepted, not on v.passed. Nothing in the schema ties the boolean to the dict. They are two independently generated fields. So this is schema-valid output:

{"reasoning": "", "criteria_met": {}, "passed": true, "issues": []}

Valid, and it would end the loop on round 1 against an empty criteria dict. accepted is the one line that makes criteria_met load-bearing: it demands the claim (passed), demands the evidence covers all five keys, and demands every one is True.

Render with as_feedback, not the raw list. issues is a list[str], and what goes into the next prompt is text. Interpolate the list into an f-string directly and the generator is shown ['add caveats', 'cite section']: a stringified Python list, where you meant two bullet points. One line to fix, and both tiers below would otherwise get it wrong the same way.

Tier 2 — LangGraph

LangGraph is a library that expresses an agent as a graph. Nodes are ordinary functions; edges are the allowed transitions between them. The framework runs the graph, carrying a shared state dictionary from node to node: each node reads the state, returns a partial update, and the framework merges that update in.

The state’s keys and their types are declared up front with a TypedDict: a Python class that describes the shape of a dictionary (which keys it has, and what type each value is) without changing the fact that it is a plain dict at runtime. The S class below is exactly that: the loop’s state, spelled out. Verdict, accepted, and as_feedback are reused from the section above.

The pattern has four parts: the two model clients, the state class S, the three functions (generate, evaluate, route), and the graph wiring that connects them.

from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langchain_anthropic import ChatAnthropic

# max_tokens is a truncation ceiling, not a budget: a reply that reaches it
# comes back cut off mid-sentence with `stop_reason: "max_tokens"`. 8192 for a
# draft, 2048 for a verdict that is four short fields.
llm = ChatAnthropic(model="claude-opus-5", max_tokens=8192)
# with_structured_output is what makes v.issues / v.passed exist — without it,
# judge.invoke returns text and you are back to parsing prose.
judge = ChatAnthropic(model="claude-opus-5", max_tokens=2048).with_structured_output(Verdict)

class S(TypedDict):
    task: str
    draft: str
    feedback: str         # text, not v.issues — see as_feedback above
    rounds: int
    passed: bool          # written by evaluate, read by route — must be declared

def generate(s: S) -> S:
    prompt = s["task"] if not s["draft"] else (
        f"{s['task']}\n\nPrevious:\n{s['draft']}\n\nFix:\n{s['feedback']}")
    return {"draft": llm.invoke(prompt).content, "rounds": s["rounds"] + 1}

def evaluate(s: S) -> S:
    v = judge.invoke(f"Task: {s['task']}\nDraft: {s['draft']}")
    return {"feedback": as_feedback(v.issues), "passed": accepted(v)}

def route(s: S) -> str:
    return END if s.get("passed") or s["rounds"] >= 3 else "generate"

g = StateGraph(S)
g.add_node("generate", generate)
g.add_node("evaluate", evaluate)
g.add_edge(START, "generate")
g.add_edge("generate", "evaluate")
g.add_conditional_edges("evaluate", route, ["generate", END])
app = g.compile()

# Seed EVERY key in S. LangGraph does not default a missing one, so
# `app.invoke({"task": "..."})` raises KeyError: 'draft' inside the first node,
# and adding only `draft` then raises KeyError: 'rounds'.
final = app.invoke({"task": "write the payments integration guide",
                    "draft": "", "feedback": "", "rounds": 0, "passed": False})
print(final["passed"], final["draft"][:80])

Four things about this block:

route is a conditional edge, not a while. That is the reason to reach for LangGraph here. The loop lives in the graph, so the retry edge evaluate → generate is a declared part of the structure and not a line of imperative code, which is what lets the framework checkpoint state between nodes, resume a run that died on round 2, and stream node-by-node progress. A while loop gets you none of that, which is why Tier 3 has to own its own state.
passed must be in S. Nodes return partial dicts that get merged into the state, so evaluate writing {"passed": ...} and route reading s.get("passed") only work if the key is declared. If it is missing from the TypedDict, route silently reads None and loops until the round cap.
route checks rounds >= 3 in the same expression as passed. The cap is part of the edge condition, not a separate guard, so no path through the graph iterates without it. This is the oscillation fix expressed as graph structure instead of as discipline.
The cap lands on six calls again. rounds is incremented by generate, and route runs after evaluate, so a cap of 3 buys three generates and three evaluates, six calls, and the last draft is judged before route sees it. Tier 3 counts rounds of judging instead and stops at the same six. Same loop, different fencepost, same bill.

One pass through the graph:

generate  rounds 0 -> 1, writes d0     route not consulted (unconditional edge)
evaluate  d0 -> fail                   route: passed False, rounds 1 < 3 -> generate
generate  rounds 1 -> 2, writes d1
evaluate  d1 -> fail                   route: passed False, rounds 2 < 3 -> generate
generate  rounds 2 -> 3, writes d2
evaluate  d2 -> fail                   route: rounds 3 >= 3 -> END, returns d2

Three generates, three evaluates, and the returned draft d2 was evaluated on the step immediately before the graph ended.

Verifying the nodes without a network

You can drive this graph against a small fake that records the calls it makes and returns scripted verdicts, so the two node functions and the router run with no model and no credentials. What such a test can and cannot prove is worth being honest about:

It proves the nodes and routing are right: that a passing verdict stops the graph after one round, that all of S must be seeded or the first node raises KeyError, that the cap halts at three generate/evaluate pairs, that feedback reaches the next prompt as text and not as a repr’d list, and that route gates on accepted so an empty criteria_met cannot end the loop.
It does not prove LangGraph works. Checkpointing, resume, and streaming, point 1’s whole argument for using a graph framework, are untested; that claim rests on the library’s documentation.
No model is called, so truncation, refusal, and empty replies are not exercised here. Tier 3 tests those, on the SDK path where the code that handles them lives.

Tier 3 — Anthropic SDK

The same loop with no framework at all, just the vendor’s Python client. Verdict, CRITERION_KEYS, accepted, and as_feedback are repeated here, unchanged, so the listing runs on its own. Read it in five parts:

The schema and helpers, copied from above.
JUDGE_SYSTEM and CRITERIA: the judge’s instructions and the five criteria in prose.
first_text: the reply reader, which is where three failure modes are caught.
generate and evaluate: one model call each.
optimize: the loop itself, in eleven lines.

from __future__ import annotations   # `list[str] | None` needs this on 3.9

import anthropic
from pydantic import BaseModel, Field

client = anthropic.Anthropic()

class Verdict(BaseModel):
    reasoning: str = Field(description="Cite specific evidence before judging.")
    criteria_met: dict[str, bool]
    passed: bool
    issues: list[str]

CRITERION_KEYS = ("api_cited", "code_runs", "errors_named",
                  "sentence_len", "no_invention")

def accepted(v: Verdict) -> bool:                  # the claim AND the evidence
    return (v.passed
            and all(k in v.criteria_met for k in CRITERION_KEYS)
            and all(v.criteria_met.values()))

def as_feedback(issues: list[str]) -> str:
    return "\n".join(f"- {i}" for i in issues)

class NotAnAnswer(RuntimeError):
    """The reply stopped for a reason that is not a finished draft."""

JUDGE_SYSTEM = (
    "You are a strict evaluator. Judge the draft against every criterion. "
    "Report each unmet criterion as one concrete, actionable issue. "
    "Do not rewrite the draft."
)

# This is what "clear, articulable criteria" actually looks like. Each line is
# independently checkable and independently reportable — that is what makes
# `criteria_met: dict[str, bool]` meaningful instead of decorative. The five
# names here are CRITERION_KEYS, and the two lists have to move together.
CRITERIA = """<criteria>
api_cited:     Every claim about API behaviour cites a section number from the spec.
code_runs:     Every code sample is valid Python 3.9 and imports only the stdlib.
errors_named:  The error-handling section names all four documented failure codes.
sentence_len:  No sentence exceeds 40 words.
no_invention:  No feature is described that does not appear in the spec.
</criteria>"""

def first_text(r) -> str:
    """The only safe way to read a reply.

    Never `r.content[0]` and never a bare `next(...)` over it: a refusal
    arrives as an ordinary HTTP 200 whose content list is EMPTY, so both
    spellings crash the loop instead of reporting a decline.
    """
    if r.stop_reason == "refusal":
        raise NotAnAnswer(f"model declined: {getattr(r, 'stop_details', None)}")
    if r.stop_reason == "max_tokens":
        raise NotAnAnswer("truncated at max_tokens: raise the ceiling or split the task")
    text = next((b.text for b in r.content if b.type == "text"), None)
    if text is None:                   # thinking blocks only, or nothing at all
        raise NotAnAnswer(f"no text block in the reply ({r.stop_reason})")
    return text

def generate(task: str, draft: str = "", issues: list[str] | None = None) -> str:
    prompt = task if not draft else (
        f"{task}\n\n<previous_draft>\n{draft}\n</previous_draft>\n"
        f"<issues>\n{as_feedback(issues or [])}\n</issues>\n"
        "Produce a revised version that resolves every issue."
    )
    r = client.messages.create(
        model="claude-opus-5",
        max_tokens=8192,                       # truncation ceiling, not a budget
        messages=[{"role": "user", "content": prompt}],
    )
    return first_text(r)

def evaluate(task: str, draft: str) -> Verdict:
    r = client.messages.parse(
        model="claude-opus-5",
        max_tokens=2048,                       # a verdict is four short fields
        system=[{                              # byte-identical every round
            "type": "text",
            "text": JUDGE_SYSTEM + "\n\n" + CRITERIA,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{"role": "user",
                   "content": f"<task>{task}</task>\n<draft>{draft}</draft>"}],
        output_format=Verdict,
    )
    return r.parsed_output

def optimize(task: str, max_rounds: int = 3) -> tuple[str, bool]:
    draft = generate(task)
    for round_no in range(1, max_rounds + 1):
        v = evaluate(task, draft)          # every draft that can be returned
        if accepted(v):                    # goes through here first
            return draft, True
        if round_no == max_rounds:
            return draft, False            # judged, and it failed. Never fake success
        draft = generate(task, draft, v.issues)
    return draft, False                    # max_rounds < 1: nothing was judged,
                                           # so this path can only ever be False

`reasoning` is the first field

Constrained decoding generates fields in declared order (structured output is a guarantee, not a request), so putting passed first would force the model to commit before reasoning. Order is not cosmetic.

Branch on `stop_reason` before you touch `content`

stop_reason is the field on a reply that says why generation stopped; content is the list of blocks the model produced. The rule: ask why generation stopped before you read what it produced, and never index into content blindly. first_text exists because three endings are not a finished draft, and two of them carry no text block at all:

`stop_reason`	What actually came back	What the naive read does
`refusal`	HTTP 200, empty content list	`r.content[0].text` raises `IndexError`; `next(b.text for b in ...)` raises `StopIteration`
`max_tokens`	A draft cut off mid-word	Hands a truncated fragment to the judge as finished work
`end_turn`, thinking only	A thinking block, no text block	Same empty-generator crash as the refusal row

A refusal is not an HTTP error but a normal 200 whose content list is empty, so the loop dies with a traceback about iteration instead of reporting a decline. And on claude-opus-5, adaptive thinking is on by default, so a reply routinely leads with a thinking block; a response truncated before it reaches any text is the same empty-generator crash again. One helper, three failures.

`max_tokens` is a truncation ceiling, not a budget

It is the one parameter here that silently changes behaviour. The model is not told about it, so it does not wrap up as it approaches the limit. It is simply cut off, and you get stop_reason: "max_tokens" with a half-finished draft. 8192 for the generator and 2048 for a four-field verdict are sized to that, and first_text is what stops a truncation from being mistaken for a draft.

`output_format` guarantees the verdict parses

Invalid output has probability zero, because the runtime never lets an invalid token be chosen. No regular expressions patching up the text afterwards, and no repair loop. It guarantees shape, though, not intent, which is the accepted point below.

The judge’s system prompt is marked for caching — and as written it will not cache

Prompt caching lets the provider store its precomputed internal state for a block of prompt text, so later requests that begin with the same bytes skip recomputing it. The cache_control marker on the system block requests that, and because the block is byte-identical every round, rounds 2 and later would read the criteria at roughly a tenth of the normal input rate (prompt caching derived).

The mechanism is right; the size is not. A cached prefix has to clear the model’s minimum, which is 512 tokens on claude-opus-5. The cached block here is about 549 characters, roughly 137 tokens at ~4 characters per token (tokens). 137 against a floor of 512, so it never caches, and there is no error: the only symptom is cache_creation_input_tokens: 0 in the usage block.

Keep the marker anyway. It costs nothing and starts paying the moment the criteria list grows to a size worth caching. But verify, don’t assume: print usage.cache_read_input_tokens on round 2, and if it is zero you are paying full price every round.

One caveat before relying on it: "ephemeral" is the short-lived cache tier, and its default time-to-live is five minutes, which three claude-opus-5 calls at high effort can genuinely exceed. A one-hour tier exists for exactly that, at a higher write price: a five-minute entry costs about 1.25× the base input rate to write, a one-hour entry about 2×.

`accepted`, not `v.passed`

Nothing in the schema ties the boolean to the dict, so {"reasoning": "", "criteria_met": {}, "passed": true, "issues": []} is a schema-valid verdict that would end the loop on round 1 having checked nothing. Gate on the claim and the evidence, and require the evidence to name all five criteria.

The failure path returns `False`, on a judged draft

Reporting success on round 3 without passing is the most dangerous bug in this pattern. Returning a draft generated after the last verdict is a quieter version of the same bug: the call is wasted and the artifact is unreviewed, though the flag stays honest. Exiting after the evaluation, not after the revision, fixes both.

Verifying the guards

Six of the seven points above are claims about what the code does when something goes wrong, and each has a corresponding test you can run by stubbing the client out, no network, no credentials:

import types

CALLS: list = []          # every API call the loop makes, in order

def block(kind: str, text: str = ""):
    b = types.SimpleNamespace(type=kind)
    if kind == "text":
        b.text = text                      # a thinking block has no .text at all
    return b

def reply(stop_reason: str, *blocks):
    return types.SimpleNamespace(stop_reason=stop_reason, content=list(blocks),
                                 stop_details=None)

def fake(replies, verdicts=()):
    """A client that hands back canned replies and records the traffic."""
    rs, vs = list(replies), list(verdicts)
    def create(**kw):
        CALLS.append("generate"); return rs.pop(0)
    def parse(**kw):
        CALLS.append("evaluate"); return types.SimpleNamespace(parsed_output=vs.pop(0))
    return types.SimpleNamespace(messages=types.SimpleNamespace(create=create, parse=parse))

def verdict(passed, met=None, issues=("add caveats",)):
    return Verdict(reasoning="cited section 4.2", passed=passed, issues=list(issues),
                   criteria_met=dict.fromkeys(CRITERION_KEYS, passed) if met is None
                                else met)

drafts = lambda n: [reply("end_turn", block("text", f"d{i}")) for i in range(n)]

# ---- a refusal is HTTP 200 with EMPTY content, not an exception you can skip
client = fake([reply("refusal")])
try:
    generate("t"); raise AssertionError("a refusal was read as a draft")
except NotAnAnswer as e:
    assert "declined" in str(e), e

# ---- the call sequence is what catches the wasted seventh call
CALLS.clear()
client = fake(drafts(9), [verdict(False, met=dict.fromkeys(CRITERION_KEYS, False))] * 9)
assert optimize("t", max_rounds=3) == ("d2", False)
assert CALLS == ["generate", "evaluate"] * 3, CALLS       # 2N, not 2N+1

# ---- `accepted`, not `v.passed`: an empty criteria_met must not pass
client = fake(drafts(1), [verdict(True, met={}, issues=[])])
assert optimize("t", max_rounds=1) == ("d0", False), "empty criteria_met passed"

# ---- and a genuine pass must still come back True, or the gate above is fake
client = fake(drafts(1), [verdict(True, issues=[])])
assert optimize("t", max_rounds=1) == ("d0", True)

Three habits make tests like these worth writing:

Check the specific message, not just that something raised. A test that only asserts “an exception happened” would let all three stop_reason branches be deleted one at a time without ever failing.
Assert the call sequence, not just the answer. CALLS == ["generate", "evaluate"] * 3 is what catches the wasted seventh call; asserting only on the returned tuple would let that regress the moment someone “simplified” the exit.
Include a passing case. Three tests that all expect False are equally satisfied by a gate that is just return False. A genuinely passing verdict that must come back True is what keeps the failure tests honest.

The step cap is the one guard that cannot be defeated: for round_no in range(1, max_rounds + 1) halts at every cap, including 0 and -1, where the loop body never runs and the function returns after a single generate with passed=False. There is no path through optimize that iterates without the cap, and no cap value that turns the honest False into a True.

The criteria are the design decision

The CRITERIA block is where the rule if a criterion can be checked in code, check it in code stops being a slogan and becomes a line-by-line decision. Two of these five criteria should never reach the model.

code_runs is ast.parse, the standard-library call that parses Python and raises on invalid syntax, plus a check of what the sample imports. sentence_len is a regular expression over the text. Both are a few lines of ordinary code, and checking them in code buys three things: they become deterministic (same input, same verdict), free (no tokens, no latency), and immune to a judge having an opinion about whether a 41-word sentence is really that bad. It also shrinks the judge’s job to the three criteria that genuinely need reading comprehension: api_cited, errors_named, no_invention.

Why the named keys make this incremental

You do not have to move all of it at once, and the criteria_met dict is what lets you move it one key at a time. The same dict can be filled by code for some keys and by the model for others. accepted reads the merged dict and cannot tell the difference:

from code:   {"code_runs": True, "sentence_len": False}
from judge:  {"api_cited": True, "errors_named": True, "no_invention": True}
merged:      all five keys present -> accepted() returns False (sentence_len)

This is also why accepted insists every key in CRITERION_KEYS is present. A key nobody filled in is a criterion nobody checked, and silently dropping it would turn a five-criterion gate into a four-criterion one. Keep the code-checked lines in the prompt anyway: the generator reads the same criteria block, and a criterion the generator never sees is one it will keep violating. You would be catching the violation in code instead of preventing it.

One consistency note

The criteria are about someone else’s code, and this file is code too. The listing targets Python 3.9, which is why from __future__ import annotations is there to make list[str] | None legal, and why code_runs asks for valid 3.9 and not 3.10. Pick one floor and let every line agree with it. Two floors in one file is how a criterion ends up passing a sample the code cannot import.

Conclusion

The loop is generate → judge on a fresh context → revise, repeating until the judge passes or a round cap stops it. The cap is what prevents endless oscillation.
Return both the last draft and a boolean saying whether it passed. Returning a draft with no verdict is the most dangerous bug in the pattern; it lets a failing draft ship as approved.
At a cap of N rounds the loop makes 2N calls. An odd call count means one draft left unread, the exit revised after the last judgement instead of before it.
Judge structured output on the evidence (criteria_met), not just the claim (passed), and require the evidence to cover every criterion.
Read replies through stop_reason before touching content: refusals and thinking-only replies carry no text block and crash a naive read.
Move any criterion that code can check out of the judge. The criteria_met dict lets you migrate one key at a time.

One line to remember: judge everything you might return, and the call count tells you the truth, an even number means every draft was read, an odd number means one slipped out unseen.