Agent Evaluation

Evaluating an agent means replacing “it seems to work” with a defensible number, and every design choice follows from one fact: the system is not deterministic and cannot be made so.

Why runs never repeat

• Requests are batched on a shared GPU; batch shape sets the reduction order.
• Float addition is non-associative, so (a+b)+c != a+(b+c) in the last bits, which shifts logits, flips a near-tied token, and diverges the whole prefix.
• temperature=0 does not fix it: divergence happens inside the logits, before temperature is consulted. Some frontier models (Claude Opus 5, Sonnet 5) reject temperature/top_p/top_k with a 400.
• Consequences: never assert on exact strings; a single run is a sample, not a measurement; gate on aggregates, not individual cases.

The N=3 arithmetic

• Majority-of-3: run each case 3x, take the verdict appearing >=2 times. P_maj = p^3 + 3p^2(1-p).
• Polarizes away from 0.5: pushes reliable cases up, broken cases down. 0.5 is the fixed point.
• 30 cases at 2% flake: 0.98^30 = 0.545 -> 45% of clean PRs go red (teams stop trusting the suite). N=3 raises per-case to 0.9988 -> 0.9988^30 = 0.965, only 3.5% false-red.
• Fixes variance (run-to-run scatter), never bias (systematic error). Use majority, never best-of-3 (that turns broken green).

Outcome vs. trajectory

• Score outcome strictly, trajectory loosely. Assert on invariants (set membership + counts), never on an exact sequence (a sample + an implementation detail).
• Useful loose metrics: tool-call count, required-tool-called (grounding), forbidden-tool-called (safety, binary), retry/error rate, time/cost.
• Every case gets 4 fields: input, expected outcome, grader, tag. must_not_call is the only field that blocks a merge on a single case.

Eval pyramid & dataset

• Layers cheap->expensive: Unit (tools/parsers, deterministic, every commit) -> Component (~50 cases) -> Integration (full agent, fixed env, ~30 cases, N=3, per PR) -> Production (sampled live traffic).
• Push checks down: an integration case $2 ($6 at N=3); moving 80% of asserts down saves ~$5,760/week. ~10 min needs concurrency (90 runs serial = 135 min).
• Sources: hand-written edge cases, mined production failures (the ones that matter), synthetic (volume only — inherits model blind spots, never the source of categories). Start with 20 hand-written; grow from production.

Graders & the LLM judge

• Prefer a code assertion (free, instant, no drift). Reach for an LLM judge only for subjective dims (tone, faithfulness).
• Put reasoning first in the schema: constrained decoding emits fields in declaration order; score-first commits before any evidence and buys zero extra forward passes.
• Discrete scales with anchored levels (1-5, never 0-100); one dimension at a time.
• Schema traps: value bounds (ge/le) are checked by pydantic after the paid call, not the decoder; cache_control is a no-op under the 512-token minimum; max_tokens is one budget for thinking + output (reasoning-first truncates into a parse failure, not a low score).

Calibration & diagnosis

• Cohen’s kappa = (p_o - p_e)/(1 - p_e); 88% raw agreement can be near-worthless (all-PASS scores 0.80 on an 80%-pass set). Also report fail-class recall — a judge can hit 0.95 agreement yet catch only half the failures.
• Gate: kappa >= 0.6 AND fail-recall >= 0.8, with the whole confidence interval clearing the bar. ~50 labels can reject a judge, not accept one; accepting needs ~480 labels. Negative kappa = a wiring bug.
• RAG diagnosis is a fixed dependency order: recall@k -> survived assembly -> grounded. A passage never retrieved cannot be fixed downstream. success = recall@k x P(correct | retrieved).

Metrics, CI & observability

• Report success rate with cost/task and p95 latency; safety violations must be zero (a count, not a %).
• Cache hit rate = read / (read + write + input) is the highest-signal metric: computable per request, a step function (a stray datetime.now() drops it 0.92 -> 0.00 silently), ~4x cost swing.
• CI gate: success >= baseline - noise AND safety == 0; N=3 majority, aggregate threshold, moving baseline over a measured noise band. Version 4 invisible artifacts: prompt hash, tool-set hash, model ID, retrieval index.
• Log stop_reason per call — max_tokens and refusal raise no exception and look like a bad answer.