Exact Match and Token F1

Once an agent or a RAG pipeline produces an answer, you need a number that says how good it was. Two metrics do most of the work in open-domain QA: exact match, which is strict and unforgiving, and token-level F1, which gives partial credit for overlapping words. Both live or die on normalization, because “The Eiffel Tower.” and “eiffel tower” are the same answer and a raw string compare would call them different. This exercise implements both from scratch, the same way the SQuAD evaluation script does.

Why normalize first

A predicted answer and a gold answer rarely match byte for byte even when they mean the same thing. Casing differs, trailing punctuation sneaks in, and filler articles (“a”, “an”, “the”) pad the text without adding meaning. Before comparing, both strings are lowercased, stripped of punctuation, cleared of articles, and squeezed to single-spaced tokens. Every metric here runs on that normalized form, so the same helper feeds both exact match and F1.

The two metrics

• Exact match is a hard 0.0 or 1.0: after normalization, do the two strings match exactly? It is the metric that rewards getting the whole answer right and nothing partial.
• Token F1 grades on overlap. Split both normalized answers into tokens, count how many tokens they share (with multiplicity — a token appearing twice in each counts twice), and combine precision (shared over predicted length) and recall (shared over gold length) into their harmonic mean. This gives partial credit when the prediction has the right words buried in extra ones, or covers only part of the gold answer.

Task

Complete qa_metrics(pred, gold):

1. Normalize both strings: lowercase, remove punctuation, drop the articles a/an/the, and collapse whitespace.
2. Compute exact_match as 1.0 if the normalized strings are identical, else 0.0.
3. Tokenize each normalized string on whitespace. Compute the multiset intersection count shared (sum of min(pred_count, gold_count) over all tokens).
4. Compute precision = shared / len(pred_tokens), recall = shared / len(gold_tokens), and f1 as their harmonic mean.
5. Handle the empty case: if either token list is empty, all three of precision, recall, and f1 are 1.0 when both are empty and 0.0 otherwise.
6. Return the dict {"exact_match", "precision", "recall", "f1"} with float values.

Example

m = qa_metrics("The Eiffel Tower.", "eiffel tower")
m["exact_match"]        # -> 1.0   (both normalize to "eiffel tower")
round(m["f1"], 3)       # -> 1.0

m = qa_metrics("Paris, France", "Paris")
m["exact_match"]        # -> 0.0   ("paris france" != "paris")
round(m["precision"], 3)  # -> 0.5   (1 shared / 2 predicted tokens)
round(m["recall"], 3)     # -> 1.0   (1 shared / 1 gold token)
round(m["f1"], 3)         # -> 0.667

m = qa_metrics("no", "")
m["exact_match"]        # -> 0.0
m["f1"]                 # -> 0.0   (gold empty, pred not)

Constraints

• Do not use any external metric library (no sklearn, no evaluate, no NLTK). Standard library only.
• Punctuation is every character in string.punctuation; removing it must not merge or split words unexpectedly (strip it, leaving whitespace to separate tokens).
• Article removal is token-level: drop standalone a, an, the, not those letters inside other words.
• F1 uses multiset (count-aware) overlap, not set overlap.
• All four returned values are floats.

Approach

Factor normalization into one helper and run both metrics off it, so exact match and F1 can never disagree about casing or punctuation. Normalization lowercases, deletes punctuation, drops standalone articles with a word-boundary regex, and collapses whitespace. Exact match is a direct comparison of the two normalized strings. F1 counts shared tokens with multiplicity via Counter intersection, turns that into precision and recall, and combines them as a harmonic mean, with an explicit branch for the empty-token case so nothing divides by zero.

Solution

import re
import string
from collections import Counter
from typing import Dict

def _normalize(text: str) -> str:
    text = text.lower()
    text = "".join(ch if ch not in string.punctuation else " " for ch in text)
    text = re.sub(r"\b(a|an|the)\b", " ", text)
    return " ".join(text.split())

def qa_metrics(pred: str, gold: str) -> Dict[str, float]:
    pred_norm = _normalize(pred)
    gold_norm = _normalize(gold)

    exact_match = 1.0 if pred_norm == gold_norm else 0.0

    pred_tokens = pred_norm.split()
    gold_tokens = gold_norm.split()

    if not pred_tokens or not gold_tokens:
        # F1 is 1.0 only when both sides are empty, else 0.0.
        both_empty = not pred_tokens and not gold_tokens
        score = 1.0 if both_empty else 0.0
        return {
            "exact_match": exact_match,
            "precision": score,
            "recall": score,
            "f1": score,
        }

    shared_counts = Counter(pred_tokens) & Counter(gold_tokens)
    shared = sum(shared_counts.values())

    if shared == 0:
        return {
            "exact_match": exact_match,
            "precision": 0.0,
            "recall": 0.0,
            "f1": 0.0,
        }

    precision = shared / len(pred_tokens)
    recall = shared / len(gold_tokens)
    f1 = 2 * precision * recall / (precision + recall)

    return {
        "exact_match": exact_match,
        "precision": precision,
        "recall": recall,
        "f1": f1,
    }

Walkthrough

Tracing qa_metrics("Paris, France", "Paris"):

1. _normalize("Paris, France") lowercases to "paris, france", replaces the comma with a space giving "paris france", finds no articles, and collapses whitespace to "paris france". _normalize("Paris") yields "paris".
2. exact_match compares "paris france" to "paris" — not equal, so 0.0.
3. pred_tokens = ["paris", "france"], gold_tokens = ["paris"]. Neither is empty.
4. Counter(pred_tokens) & Counter(gold_tokens) is {"paris": 1}, so shared = 1.
5. precision = 1 / 2 = 0.5, recall = 1 / 1 = 1.0, f1 = 2 * 0.5 * 1.0 / 1.5 = 0.667.

For qa_metrics("The Eiffel Tower.", "eiffel tower"), both normalize to "eiffel tower": exact match is 1.0, the two token lists are identical, shared = 2, and precision, recall, and f1 are all 1.0. For qa_metrics("no", ""), gold normalizes to "" with no tokens while pred has one, so the empty branch returns 0.0 for precision, recall, and f1.

Complexity & notes

• Time is O(n) in the total number of characters/tokens across both strings — normalization is a single pass and the Counter intersection is linear in the token counts. Space is O(n) for the normalized strings and the two counters.
• The multiset intersection (Counter & Counter, which takes the minimum count per key) is what makes F1 count-aware. A set-based overlap would over-credit a prediction of "the" against a gold "the the"; the harmonic mean punishes both under- and over-generation, which is why F1 beats a raw overlap ratio.
• Order of normalization matters: strip punctuation before removing articles, and replace punctuation with a space rather than the empty string, so "cats,dogs" becomes two tokens instead of one glued "catsdogs". The \b(a|an|the)\b boundaries ensure only standalone articles are dropped — "another" and "cathedral" survive intact.
• The shared == 0 early return is not strictly required (the formula would give f1 = 0.0 anyway since precision and recall are both 0, but 2 * 0 * 0 / (0 + 0) would raise ZeroDivisionError), so guarding it is a correctness fix, not just an optimization.
• This is single-example scoring. A full eval averages exact_match and f1 over a dataset, and for multi-reference QA you take the max F1 over the gold answers per question before averaging.