pass@k — AI Agent Coding — InterviewPrepKit

Solving tips

Do not estimate pass@k by sampling k of the n attempts and checking if any passed — that is high-variance and biased for small n. There is a closed form: the chance all k draws miss is comb(n-c, k) / comb(n, k), so pass@k is one minus that.
The failure combinatorics only make sense when there are at least k failures to choose from. When n - c < k every k-subset must contain a passing sample, so pass@k for that task is exactly 1.0 — special-case it before calling comb.
pass@k is a per-task quantity; the reported number is the average across tasks. Keep those two steps separate so an empty suite or a single bad task is easy to reason about.

When you sample a code-generating agent many times per problem, the honest quality metric is not “did the single best sample pass” but “if a user had drawn k samples, how likely is at least one to pass.” That is pass@k. Estimating it by literally drawing k of your n attempts is noisy and biased downward, so the standard defines an unbiased closed-form estimate from the counts alone. This exercise implements that estimator.

The estimator

For a single task you sampled n attempts and c of them passed. Draw k of the n attempts uniformly at random. The draw fails only when all k come from the n - c failing attempts, which happens with probability comb(n - c, k) / comb(n, k). So the per-task pass@k is:

1 - comb(n - c, k) / comb(n, k)

This is the unbiased estimator from the HumanEval / Codex evaluation. The reported pass@k for a benchmark is the mean of this quantity over all tasks.

The one edge to watch: if n - c < k there are not even k failing samples to choose, comb(n - c, k) is 0, and the task’s pass@k is 1.0. math.comb already returns 0 when the second argument exceeds the first, but handling it explicitly keeps the intent obvious and avoids relying on that behavior.

Task

Complete pass_at_k(results, k):

If results is empty, return 0.0.
For each (n, c) task, compute the per-task estimate. If n - c < k, the estimate is 1.0; otherwise it is 1 - comb(n - c, k) / comb(n, k).
Return the average of the per-task estimates across all tasks.

Use math.comb. Do not approximate by sampling.

Example

results = [
    (5, 3),    # 5 attempts, 3 passed
    (5, 0),    # 5 attempts, none passed
    (10, 10),  # all 10 passed
    (4, 1),    # 4 attempts, 1 passed
]

# per task at k=2:
#   (5,3):  1 - comb(2,2)/comb(5,2) = 1 - 1/10  = 0.9
#   (5,0):  1 - comb(5,2)/comb(5,2) = 1 - 10/10 = 0.0
#   (10,10): n-c = 0 < 2            -> 1.0
#   (4,1):  1 - comb(3,2)/comb(4,2) = 1 - 3/6   = 0.5

round(pass_at_k(results, k=2), 4)   # -> 0.6
round(pass_at_k(results, k=1), 4)   # -> 0.4625  (mean of 0.6, 0.0, 1.0, 0.25)

Constraints

n >= 1 and 0 <= c <= n for every task; 1 <= k <= n.
Use integer combinatorics (math.comb), not floating-point factorials, to avoid overflow and precision loss for large n.
Treat each task independently; the result is a plain average (every task weighted equally).
Return 0.0 for an empty results list.

Approach

pass@k for one task is the probability that a random k-subset of its n samples is not entirely failures, which is 1 - comb(n - c, k) / comb(n, k). We compute that per task, special-casing n - c < k (fewer than k failing samples, so every k-subset holds a pass and the value is 1.0), and average across tasks. Using math.comb keeps the arithmetic exact integer combinatorics, so nothing overflows or drifts the way a factorial-ratio in floats would.

Solution

from math import comb
from typing import List, Tuple

def pass_at_k(results: List[Tuple[int, int]], k: int) -> float:
    if not results:
        return 0.0

    total = 0.0
    for n, c in results:
        if n - c < k:
            # fewer than k failing samples: every k-subset contains a pass
            total += 1.0
        else:
            total += 1.0 - comb(n - c, k) / comb(n, k)

    return total / len(results)

Walkthrough

Trace the four example tasks at k=2:

(5, 3) — n - c = 2, which is not < 2, so we use the formula: comb(2, 2) = 1, comb(5, 2) = 10, giving 1 - 1/10 = 0.9.
(5, 0) — n - c = 5: comb(5, 2) = 10 over comb(5, 2) = 10, so 1 - 1 = 0.0. All samples failed, so no k-draw can pass.
(10, 10) — n - c = 0 < 2, so the special case fires and contributes 1.0. Every attempt passed; any draw passes.
(4, 1) — n - c = 3: comb(3, 2) = 3 over comb(4, 2) = 6, so 1 - 3/6 = 0.5.

The sum is 0.9 + 0.0 + 1.0 + 0.5 = 2.4, divided by 4 tasks gives 0.6. At k=1 each per-task estimate reduces to c/n (since 1 - comb(n-c, 1)/comb(n, 1) = 1 - (n-c)/n = c/n), giving 0.6, 0.0, 1.0, 0.25, whose mean is 1.85 / 4 = 0.4625.

Complexity & notes

Time is O(T · M(n)) for T tasks, where M(n) is the cost of one comb on values up to n; space is O(1) beyond the input.
Why the closed form instead of sampling: the naive estimator (draw k of the n attempts, check if any pass, repeat) is unbiased in expectation but has real variance at small n, and people often instead compute 1 - (1 - c/n)**k, which is biased. The combinatorial form is the exact unbiased estimator used in the Codex/HumanEval paper.
The n - c < k guard is not just cosmetic: it states the intent and shields against any environment where you would rather not depend on comb returning 0 for k > n - c. math.comb does return 0 there, so even without the guard the math is correct — but the branch documents why.
Keep the arithmetic in integers until the final division. Computing comb via n! / (k!(n-k)!) in floats overflows and loses precision for large n; math.comb uses exact integer arithmetic and only the ratio becomes a float.
This is a plain (unweighted) average over tasks, matching how benchmarks report pass@k. If tasks had different sample counts and you wanted a pooled estimate you would still average the per-task estimates, not the raw counts — each task’s estimator is already the right per-task quantity.