Building an Evaluator-Optimizer Loop

An evaluator-optimizer is a draft-and-critique loop: generate a draft, judge it against pre-written criteria on a fresh context, revise on failure, and stop when the judge passes or a round cap is hit.

The loop

• Draft: one LLM call generates an answer.
• Verdict: a second call judges the draft against written criteria.
• Revise: on a failing verdict, the generator revises using the judge’s specific issues.
• Repeat until pass, or until the round cap (a hard iteration limit you set) stops it.
• Judge runs with fresh context (only task + draft, no prior conversation) so it critiques the text cold instead of defending output it just wrote.

draft = generate(task)
for round in 1..MAX:
    verdict = evaluate(task, draft)        # fresh context, structured output
    if verdict.passed: return draft, True
    if round == MAX:   return draft, False # judged and failed: report the miss
    draft = generate(task, draft, verdict.issues)

Input and output

• In: a task in plain text, and a written list of criteria the judge grades against.
• Out: the last draft (not the best; nothing ranks or scores drafts) AND a boolean saying whether it passed.
• Returning a draft with no verdict is the most dangerous bug: a failing draft ships as approved.

Call count

• 2N calls at a cap of N rounds: N generates + N evaluates. Six at N = 3.
• One generate runs before the loop; each round runs one evaluate, each round except the last runs one revising generate.
• The design chapter’s 2N+1 (seven) counts a different loop that revises on the way out, producing a draft d3 no evaluator saw.
• Rule: even call count = every draft was judged; odd count = one draft slipped out unread.

The verdict schema (pydantic)

• Fields, in declared order: reasoning (str), criteria_met (dict[str, bool]), passed (bool), issues (list[str]).
• reasoning comes first: constrained decoding generates fields in order, so evidence must exist before the verdict. passed first would commit the model before reasoning.
• Gate on accepted(v), not v.passed: schema does not tie the boolean to the dict, so {"criteria_met": {}, "passed": true} is valid and would pass having checked nothing. accepted requires passed, all CRITERION_KEYS present, and every value True.
• Render feedback with as_feedback (joins issues into bullet text); interpolating the raw list[str] shows the generator a stringified Python list.

Three tiers, same loop

• LangGraph: nodes are functions, edges are transitions, a shared TypedDict state passes between them. passed must be declared in state or route reads None and loops to the cap. The rounds >= 3 check sits in the edge condition, so no path skips the cap. Buys checkpointing, resume, streaming that a while loop cannot.
• SDK: no framework, owns its own loop and state.

SDK gotchas

• Read replies via first_text, never r.content[0] or a bare next(...). Check stop_reason before touching content.

• max_tokens is a truncation ceiling, not a budget: model is not told, just cut off. 8192 for a draft, 2048 for the four-field verdict.
• output_format (structured output) guarantees shape via constrained decoding, not intent — hence still gate on accepted.
• Prompt caching: cache_control marks the byte-identical judge system prompt, but the block (~137 tokens) is below the 512-token minimum, so it never caches. Symptom: cache_creation_input_tokens: 0. Keep the marker (costs nothing, pays once criteria grow); verify with usage.cache_read_input_tokens on round 2. "ephemeral" tier default TTL is 5 min (write cost ~1.25× base); a 1-hour tier exists (~2×).

Criteria are the design decision

• Rule: if a criterion can be checked in code, check it in code. code_runs → ast.parse + import check; sentence_len → a regex. Both become deterministic, free, and immune to judge opinion.
• The criteria_met dict lets you migrate keys one at a time: some filled by code, some by the model, accepted reads the merged dict and cannot tell the difference.
• accepted insists every CRITERION_KEYS key is present, so a dropped key does not silently shrink the gate. Keep code-checked lines in the prompt too, so the generator sees them and stops violating them.
• Keep one language-version floor (the listing targets Python 3.9, hence from __future__ import annotations for list[str] | None); two floors let a criterion pass a sample the code cannot import.

Testing without a network

• Stub the client to record calls and return canned replies.
• Check the specific message, not just that something raised (else all three stop_reason branches can be deleted one by one).
• Assert the call sequence (CALLS == ["generate","evaluate"]*3), which catches the wasted seventh call.
• Include a passing case that must return True, or a gate of return False passes every failure test.
• The for round_no in range(1, max_rounds+1) cap halts at every value including 0 and -1, and no cap turns an honest False into True.