LLM-as-Judge Eval Harness — AI Agent Coding

Solving tips

The judge is a model, so treat its output as untrusted: clamp the score into [0, 1] and never let a malformed or crashing judge take down the run. A bad grade is still a data point.
Separate the two numbers reviewers care about: the mean score tells you overall quality, the flagged list tells you which specific cases to go look at. One without the other is not actionable.
Be ready to explain why you'd use an LLM judge at all (open-ended outputs a substring check can't grade) and its failure modes (position/verbosity/self bias, and that it needs an explicit rubric to be reproducible).

A substring check can tell you whether an agent said “Paris,” but it cannot tell you whether a summary is faithful, whether an explanation is helpful, or whether a rewrite kept the original meaning. Those are open-ended judgments, and the practical way to score them at scale is to ask another model to grade the output against a rubric. That grader is the LLM judge, and this exercise is the harness that runs it over a batch and turns its opinions into numbers you can track.

Why an LLM judge

Rule-based checks are cheap, deterministic, and perfect when correctness is a predicate. They fall apart on free-form text where many different strings are all “good.” An LLM judge fills that gap: given the prompt and the output, it returns a graded score and a short reason. The cost is that the judge is itself a model, with all the fallibility that implies, so the harness has to defend against it rather than trust it.

How a case works

A case is a dict with a prompt (what the agent was asked) and an output (what it produced). The harness packs those into a pair dict and hands it to judge, which returns {"score": <0..1>, "reason": <str>}. You record the clamped score and its reason, decide whether the score falls below threshold, and move on. Because judge is passed in, swapping in a deterministic stub makes the whole harness testable without a network.

Task

Complete run_judge_eval(judge, cases, threshold=0.5):

Iterate the cases in order. For each, resolve its name (default "case_<i>" using the zero-based index).
Build pair = {"prompt": case["prompt"], "output": case["output"]} and call judge(pair).
Read the score and clamp it into [0.0, 1.0] as a float; keep the reason (default ""). If judge raises or the score is not numeric, the case scores 0.0, error holds the exception text, and reason records it too.
A case is flagged when its (clamped) score is strictly less than threshold.
Collect one result dict per case: {"name", "score", "reason", "flagged", "error"} (error is None on success).
Return the summary with results, mean_score (mean of scores, or 0.0 when empty), flagged (the names below threshold, in order), and total.

Example

def judge(pair):
    # toy judge: full marks when the gold answer appears in the output
    gold = {"2+2": "4", "cap": "Paris"}
    want = gold.get(pair["prompt"], "")
    if want and want in pair["output"]:
        return {"score": 1.0, "reason": "contains the expected answer"}
    return {"score": 0.0, "reason": "missing the expected answer"}

cases = [
    {"name": "math", "prompt": "2+2", "output": "the answer is 4"},
    {"name": "geo",  "prompt": "cap", "output": "the capital is Paris"},
    {"name": "bad",  "prompt": "2+2", "output": "i don't know"},
]

summary = run_judge_eval(judge, cases, threshold=0.5)

round(summary["mean_score"], 3)              # -> 0.667
summary["flagged"]                           # -> ["bad"]
[r["score"] for r in summary["results"]]     # -> [1.0, 1.0, 0.0]
summary["results"][2]["flagged"]             # -> True

Constraints

One case whose judge crashes or returns garbage must not stop the run; it scores 0.0 and every case still gets a result.
Clamp every score into [0.0, 1.0]; a judge that returns 1.4 or -0.2 must not skew the mean.
Preserve case order in both results and flagged.
Do not call any real LLM, network, or clock — judge is provided and is the only grader.

Approach

The harness is a loop that packs each case into a pair, calls the injected judge, and defends against everything the judge might do wrong. We clamp the returned score into [0, 1] and treat a raised exception or a non-numeric score as a 0.0 grade with the error recorded, so one broken grade never sinks the run. After the loop we compute the mean of the scores and collect the names of every case that fell below the threshold, giving reviewers both the aggregate quality signal and the specific cases to inspect.

Solution

from typing import Any, Callable, Dict, List

def run_judge_eval(
    judge: Callable[[Dict[str, str]], Dict[str, Any]],
    cases: List[Dict[str, str]],
    threshold: float = 0.5,
) -> Dict[str, Any]:
    results: List[Dict[str, Any]] = []
    scores: List[float] = []
    flagged: List[str] = []

    for i, case in enumerate(cases):
        name = case.get("name", f"case_{i}")
        pair = {"prompt": case["prompt"], "output": case["output"]}
        error = None

        try:
            verdict = judge(pair)
            score = float(verdict["score"])          # may raise on non-numeric
            score = max(0.0, min(1.0, score))         # clamp into [0, 1]
            reason = verdict.get("reason", "")
        except Exception as e:                        # untrusted grader: absorb failures
            score = 0.0
            reason = f"judge error: {e}"
            error = str(e)

        is_flagged = score < threshold
        if is_flagged:
            flagged.append(name)

        scores.append(score)
        results.append({
            "name": name,
            "score": score,
            "reason": reason,
            "flagged": is_flagged,
            "error": error,
        })

    mean_score = sum(scores) / len(scores) if scores else 0.0

    return {
        "results": results,
        "mean_score": mean_score,
        "flagged": flagged,
        "total": len(cases),
    }

Walkthrough

On the example, threshold=0.5:

math: pair={"prompt": "2+2", "output": "the answer is 4"}. The stub judge finds "4" in the output and returns {"score": 1.0, ...}. Clamped to 1.0, 1.0 < 0.5 is false, so not flagged.
geo: the judge finds "Paris" and returns 1.0. Not flagged.
bad: the judge misses "4" in "i don't know" and returns 0.0. 0.0 < 0.5 is true, so "bad" is appended to flagged and results[2]["flagged"] is True.
After the loop, mean_score = (1.0 + 1.0 + 0.0) / 3 = 0.667, flagged == ["bad"], total == 3.

Had the judge raised on some case, the except branch would score it 0.0, set reason="judge error: ..." and error to the message, and the case would still appear in results and (being below threshold) in flagged.

Complexity & notes

Time is O(n × judge cost) for n cases; the aggregation is a single O(n) pass. Space is O(n) for results, scores, and flagged.
Why an LLM judge: open-ended outputs (summaries, explanations, rewrites) have many valid surface forms, so a substring or regex check either passes everything or nothing. A judge model scores against a rubric the way a human rater would, which is the only scalable option for these tasks.
Pitfalls to name in an interview: judges carry biases — position bias (favoring the first option in a pairwise compare), verbosity bias (longer looks better), and self-preference (a model rating its own family higher). They are also non-deterministic and drift across model versions. The mitigations are an explicit written rubric, a low or zero temperature, randomizing option order in pairwise setups, and periodically calibrating the judge against human labels.
Clamping and error absorption are load-bearing: the judge is a model, so its output is untrusted input. Without the clamp a stray 1.4 inflates the mean; without the try/except one malformed verdict aborts a long, expensive run. Recording the error text keeps those failures visible instead of silently scoring 0.0 for an unknown reason.
A natural extension is to return the full distribution (min, median, percentiles) rather than only the mean, since a single average hides a bimodal split where half the cases are excellent and half are broken.