A Rule-Based Eval Harness

Before you trust an agent, you have to measure it, and the smallest useful measurement is a rule-based eval: a fixed set of inputs, each paired with a check that decides pass or fail. This exercise is the harness that runs those checks and reports a score. It is the scaffolding every larger eval suite is built on, so the control logic has to be boringly reliable.

How a case works

A case is a dict describing one test. It always carries an input to feed the agent, and exactly one kind of check:

• A check function — case["check"] is a callable that takes the agent’s output string and returns True for a pass. Use this when correctness is a predicate (length, format, a parsed value, an exact match).
• An expected substring — case["expected"] is a string that must appear somewhere in the output. This is the common “did it mention the answer” check.

You run agent_fn(case["input"]) to get the output, then apply whichever check the case carries. The agent is passed in, so the same cases can grade any implementation, and swapping in a stub agent makes the harness deterministic and testable on its own.

Task

Complete run_eval(agent_fn, cases):

1. Iterate the cases in order. For each, resolve its name (default "case_<i>" using the zero-based index).
2. Call agent_fn(input). If it raises, the case fails with output=None and the exception text in error.
3. Apply the check. A "check" case passes when bool(check(output)) is true; an "expected" case passes when the substring is in the output. If the check itself raises, the case fails.
4. Collect one result dict per case: {"name", "passed", "output", "error"} (error is None on success).
5. Return the summary dict with results, passed, total, and pass_rate (passed / total, or 0.0 when there are no cases).

Example

def agent(x):
    # toy agent: echoes a canned answer for known inputs
    return {"2+2": "the answer is 4", "cap": "Paris is the capital"}.get(x, "i don't know")

cases = [
    {"name": "math",     "input": "2+2", "expected": "4"},
    {"name": "geo",      "input": "cap", "check": lambda o: "Paris" in o},
    {"name": "unknown",  "input": "???", "expected": "42"},
]

summary = run_eval(agent, cases)

summary["passed"]                       # -> 2
summary["total"]                        # -> 3
round(summary["pass_rate"], 3)          # -> 0.667
[r["passed"] for r in summary["results"]]  # -> [True, True, False]
summary["results"][2]["output"]         # -> "i don't know"

Constraints

• One failing or crashing case must not stop the run; every case gets a result.
• Assume each case has exactly one of check or expected; if both are somehow present, prefer check.
• Preserve case order in results.
• Do not call any real LLM, network, or clock — agent_fn and the checks are all provided.

Approach

The harness is a single pass over the cases. For each case we run the agent inside a try so a crashing agent becomes a failing case, resolve the check kind (a callable predicate or an expected substring), and apply it inside its own try so a crashing check also just fails the case. We collect a uniform result dict per case and then compute the aggregate pass_rate from the tally, guarding the empty-suite divide-by-zero.

Solution

from typing import Any, Callable, Dict, List

def run_eval(
    agent_fn: Callable[[Any], str],
    cases: List[Dict[str, Any]],
) -> Dict[str, Any]:
    results = []

    for i, case in enumerate(cases):
        name = case.get("name", f"case_{i}")
        output = None
        error = None
        passed = False

        # 1. run the agent; a crash is a failing case, not a crashing harness
        try:
            output = agent_fn(case["input"])
        except Exception as e:
            error = f"agent error: {e}"
            results.append({"name": name, "passed": False,
                            "output": None, "error": error})
            continue

        # 2. apply the check; a crashing check also just fails the case
        try:
            if "check" in case:
                passed = bool(case["check"](output))
            else:
                passed = case["expected"] in output
        except Exception as e:
            passed = False
            error = f"check error: {e}"

        results.append({"name": name, "passed": passed,
                        "output": output, "error": error})

    total = len(cases)
    n_passed = sum(1 for r in results if r["passed"])
    pass_rate = n_passed / total if total else 0.0

    return {
        "results": results,
        "passed": n_passed,
        "total": total,
        "pass_rate": pass_rate,
    }

Walkthrough

Trace the three example cases against the toy agent:

1. math — agent("2+2") returns "the answer is 4". The case has expected="4"; "4" in "the answer is 4" is True, so passed=True, error=None.
2. geo — agent("cap") returns "Paris is the capital". The case has a check, so we evaluate bool(("Paris" in o)) on that output, which is True. Because "check" in case, the check branch wins even if an expected were also present.
3. unknown — agent("???") hits the .get default and returns "i don't know". The case has expected="42"; "42" in "i don't know" is False, so passed=False with the output still recorded for inspection.

The tally is n_passed=2, total=3, pass_rate=2/3≈0.667, and [r["passed"] for r in results] is [True, True, False] in original order.

Complexity & notes

• Time is O(n × cost of one agent call plus one check); the harness bookkeeping is O(1) per case. Space is O(n) for the results list (plus whatever the outputs weigh).
• Wrapping the agent call and the check in separate try blocks is deliberate: it lets you distinguish an agent that threw ("agent error: ...") from a check that threw ("check error: ..."), which matters when you are debugging which side is broken.
• Recording output on failures — not just the boolean — is what turns a red score into an actionable diff. An eval that only reports pass/fail forces you to re-run to see what the agent actually said.
• Guard the empty suite: pass_rate is defined as 0.0 when total == 0 so the harness never raises ZeroDivisionError on an empty case list.
• Preferring check when both keys are present makes the resolution rule total and order-independent; in an interview, state the tie-break explicitly rather than leaving it to dict iteration.
• This skeleton generalizes cleanly: add a weight per case for a weighted rate, group results by a tag for per-category breakdowns, or run cases concurrently since each is independent — the per-case isolation here is exactly what makes that safe.