Model Debugging Playbook

A model is underperforming and time is limited. What do you do first?

In this lesson, we’ll work through a seven-step procedure. Step 0 reproduces the number you are unhappy about. Step 6 trains a better model. Do not reach step 6 until the six cheaper explanations in between have been ruled out.

By the end you’ll be able to turn a complaint as vague as “the model is bad” into three concrete things: a named broken assumption, a number that proves the assumption is broken, and the one experiment that would confirm or refute it.

The organizing idea is that a bug is an assumption you did not know you were making. Every step below opens by naming the assumption it puts on trial.

The order is the whole point. Each step either eliminates a whole class of explanation or produces the number the next step needs. Run them out of order and you can spend days tuning a step size on a pipeline whose labels are shuffled. (A hyperparameter is a setting you choose before training instead of learning from data: how large a step the training algorithm takes each time it adjusts the model, for instance.)

What goes in, and what comes out

Two separate input/output pairs are in play, and confusing them is a common source of wasted time: one pair belongs to the model, the other to the playbook.

What goes into the model, and what comes out

A supervised model is a function fitted to past examples.

• Feature vector: the input. One row of measurable properties describing one thing: a user, a transaction, an image. A churn model’s row might be purchases_7d = 3 (purchases in the last seven days) and tenure_days = 88.
• Prediction: the output. A class (“churned” or “retained”), a number, or most often a score between 0 and 1 that is meant to behave like a probability.
• Label: the correct answer for that row, recorded by a human annotator or emitted by a business process.

Rows get divided into three disjoint sets, and telling them apart matters in every step below.

Four more words describe how training itself runs.

• Batch: the group of rows processed together in one update.
• Loss: a single number measuring how wrong the model’s outputs are on a batch.
• Gradients: the per-parameter directions that would reduce that loss fastest.
• Generalization: how well the model does on rows it never saw.

Two words say where a number was measured. Offline means measured on stored rows. Online means measured on live traffic in production. A large part of this chapter is about the two disagreeing.

What goes into the playbook, and what comes out

Its input is a symptom plus four artifacts.

The symptom is one of these: an offline number that looks wrong, an online number that disagrees with the offline one, or a metric that improved while the business did not.

The four artifacts are the training code, the data and its splits, the serving code path, and the logs of what production actually scored. Most steps below are unrunnable without all four, so if one is missing, note it before you start.

Its output is a single sentence in this shape:

Assumption X is false. Here is the number that proves it. Here is the fix. Here is the check that would have failed if I were wrong.

A debugging session that ends in “we tried some things and it got a bit better” has produced no such sentence and will not survive its next repetition.

The measurement words

Six words for scoring a model. AUC gets its own explanation below.

AUC is short for area under the receiver operating characteristic (ROC) curve. The name is misleading; read it as a probability, not as an area:

AUC is the chance that the model gives a randomly chosen positive row a higher score than a randomly chosen negative row.

So 0.5 is a coin flip and 1.0 is perfect ranking. See ROC-AUC as a probability for the derivation.

This lesson is the applied capstone of the machine-learning track. Diagnosing a training run diagnoses a training run from its loss curve. Here we diagnose a model from its metrics, and the surface we search is much larger.

The procedure

The whole playbook fits in one flowchart, read top to bottom: diamonds are questions you answer with a measurement, boxes are conclusions. Red boxes are terminal states reached because something is broken; orange is a real limit that is nobody’s bug; green is a step you reach only once the broken states are excluded.

flowchart TD
    S0["STEP 0<br/>Reproduce · seed everything · baseline"] --> G0{"Beats the<br/>baseline?"}
    G0 -->|no| STOP["The model adds nothing.<br/>Nothing below matters yet."]
    G0 -->|yes| S1{"STEP 1a<br/>Can it overfit<br/>8 samples to ~0 loss?"}

    S1 -->|no| WIRE["Wiring bug: label alignment,<br/>loss axis, frozen params"]
    S1 -->|yes| S1A{"STEP 1b<br/>Blind relabel of 100 rows<br/>agrees with the stored label?"}
    S1A -->|no| NOISE["LABEL NOISE<br/>caps every metric below"]
    S1A -->|yes| S1B{"Offline number<br/>implausibly high?"}
    S1B -->|yes| LEAK["LEAKAGE<br/>single-feature AUC scan<br/>then ablate"]
    S1B -->|no| S1C{"Offline good,<br/>online bad?"}
    S1C -->|yes| SKEW["TRAIN/SERVE SKEW<br/>replay serving logs<br/>through the offline scorer"]
    S1C -->|no| S2{"STEP 2<br/>Learning curve shape?"}

    S2 -->|"converged, high error"| BIAS["BIAS-limited<br/>too simple for the pattern"]
    S2 -->|"gap, val still falling in n"| VAR["VARIANCE-limited<br/>more data or more regularization"]
    S2 -->|"converged, low error"| S3["STEP 3<br/>Error analysis by slice"]
    VAR --> S3
    BIAS --> S3

    S3 --> G3{"One slice<br/>catastrophic?"}
    G3 -->|yes| FIXS["Fix the slice:<br/>feature availability, sampling,<br/>slice-specific data"]
    G3 -->|no| S4{"STEP 4<br/>Does the metric<br/>match the decision?"}
    S4 -->|no| METRIC["Fix the metric<br/>or the operating point"]
    S4 -->|yes| S5{"STEP 5<br/>Are the labels right?"}
    NOISE --> S5
    S5 -->|no| LBL["Estimate the noise ceiling.<br/>Relabel TEST first."]
    S5 -->|yes| S6["STEP 6<br/>Capacity first, then features,<br/>then data — in that order"]

    style STOP fill:#9d0208,color:#fff
    style WIRE fill:#9d0208,color:#fff
    style LEAK fill:#9d0208,color:#fff
    style SKEW fill:#9d0208,color:#fff
    style NOISE fill:#bc6c25,color:#fff
    style S3 fill:#2d6a4f,color:#fff
    style S6 fill:#2d6a4f,color:#fff

Every branch that terminates early terminates on a bug, and the only path that reaches “train a better model” is the one where six other explanations have been ruled out. In applied work, most underperformance is a data, label, metric, or skew problem, and model improvements are the residual.

Step 0 — Reproduce, seed, baseline

A measurement means something only after three things are done: every source of randomness is fixed, the metric’s variation when nothing changes is known, and the number is anchored against a trivial model.

Assumption on trial: the number I just measured is the number I would measure again tomorrow, and it means something on its own. Both halves are usually false. Randomness moves the number, and without a floor to compare against, the number has no scale.

Seed everything, including the split

A seed is the starting value of a pseudorandom number generator. Fixing it makes every “random” choice repeat identically on the next run. The helper below fixes every source of randomness that is known to move a metric; the trailing comment block lists the ones it cannot fix for you, because they live in your own data code.

import os, random
import numpy as np
import torch

def seed_everything(seed: int = 0) -> None:
    """Every source of randomness that can move a metric."""
    os.environ["PYTHONHASHSEED"] = str(seed)      # dict/set iteration order
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)                       # covers CUDA too in recent versions
    torch.backends.cudnn.deterministic = True     # pick deterministic kernels
    torch.backends.cudnn.benchmark = False        # autotuner picks by timing -> nondeterministic
    # And the ones people forget:
    #   - the train/val/test split seed
    #   - the DataLoader worker_init_fn (each worker reseeds independently)
    #   - augmentation RNG
    #   - any hashing-based feature (hash bucket assignment)

A few of those lines need translation. CUDA is the interface through which code runs on an NVIDIA graphics processor and cuDNN is its neural-network library; cuDNN ships several interchangeable implementations of each operation and by default picks between them by timing them, which depends on what else the machine is doing. benchmark = False switches that off. Augmentation is the practice of randomly perturbing training inputs (cropping, flipping, noise) to enlarge the dataset; it is a second generator that needs its own seed. A DataLoader worker is a background process that prepares batches, and each reseeds itself independently unless told otherwise. A hashing-based feature maps a category name to one of a fixed number of buckets by hashing it; change the hash seed and every category lands in a different bucket.

The split seed matters most. The standard error of a measurement is how far it moves purely from which rows happened to be sampled. For a proportion such as accuracy it is sqrt(p*(1-p)/n); at a test set of n = 2,000 and p = 0.90 that is about 0.67 percentage points. So two otherwise-identical runs that differ only in which rows landed in the test set typically land about two thirds of a point apart. An unseeded split means every experiment scores a different test set, so a “0.4 point improvement” is smaller than the noise from re-splitting. You are measuring which rows landed where, not your change.

Establish the noise floor first

Before comparing two configurations you need to know how far apart two identical configurations land. Run the same config five times, changing only the seed:

seed   val AUC
  0     0.912
  1     0.907
  2     0.918
  3     0.903
  4     0.914
        -----
mean    0.9108     sd 0.0059

Here val is validation and sd is the standard deviation, the spread. But sd = 0.0059 is the spread of one measurement, and you never decide on one measurement. You decide on a difference: config A’s score minus config B’s. Subtracting two independent draws adds their variances, so the spread of a difference is sd * sqrt(2) = 0.0083, larger than the spread of a single run.

That gap has teeth. Applying the familiar “two standard deviations” rule with the single-run sd gives a band of 2 * 0.0059 = 0.0118, which sits only sqrt(2) standard deviations out on the spread that actually governs a difference, and a sqrt(2)-sigma threshold lets through about 15.7% of pure noise. You believed you built a 5% filter; you built a 1-in-6 one. The sqrt(2) cancels the sd, so this mistake always lets through 15.7%, on every dataset. It is the same one-variance-where-two-are-needed error that two-sample testing exists to avoid (A/B testing).

Which band you may actually use depends on how many runs you paid for.

The 1.96 is the number of standard deviations that contains 95% of a normal distribution; averaging 5 runs shrinks each side’s spread by sqrt(5). The small band costs ten training runs instead of two. Fix whichever band you earned and hold every later experiment to it. Many apparent weekly “improvements” fall inside it.

The baseline, and why skipping it poisons everything after

A baseline gives your metric a scale. Build three, cheapest first.

Prevalence is how common the positive class is; a majority-class baseline always predicts the more common class, so beating it proves your metric measures something other than that imbalance. A linear model predicts by adding up the features with one weight each (logistic regression squashes that sum into a probability), and a regularized one adds a penalty on large weights to discourage memorizing noise. A GBDT (gradient-boosted decision tree) is an ensemble of small trees added one at a time, each correcting the previous ones’ errors, still the strongest cheap model on tabular data (why GBDTs still beat neural nets on tabular data).

Here is the failure all three prevent, on a churn task (predicting who will leave within 30 days):

class balance:  94.2% retained, 5.8% churned

deep model accuracy .............. 0.940
"predict retained" accuracy ...... 0.942     <- the constant function wins
logistic regression, 6 features ... AUC 0.911
deep model ....................... AUC 0.926

Two facts fall out. First, accuracy was the wrong metric: a constant “retained” scores 0.942 because 94.2% of customers are retained (that is step 4’s territory). Second, the model’s real contribution is 0.926 - 0.911 = 0.015 AUC over six features and a one-line logistic regression, which may or may not be worth its serving cost, but is now a decision instead of an assumption. Without a baseline you cannot tell “good” from “the problem was easy” from “the metric is degenerate.” The baseline also serves as a control: one trained on the same pipeline moves when the pipeline moves, so if it shifts after a change nobody thought risky, the pipeline broke.

Step 1 — Data bug or model bug?

Separating a broken pipeline from a weak model takes four checks, all cheap enough to run in an afternoon, each cheaper than the next and each ruling out more.

Assumption on trial: the data the model sees is the data I think it sees. Every check below is a different way for that to be false: the inputs are wired to the wrong labels, the labels are wrong, the production inputs differ from the training inputs, or the inputs secretly contain the answer.

1a. Overfit a single batch

This check asks whether the training machinery works at all, independent of whether the model is any good.

Overfitting is fitting the training rows so closely that the model absorbs their noise as well as their pattern, so training error keeps falling while validation error stops falling or rises. Everywhere else in this chapter that is a failure (step 2’s variance-limited branch is exactly this). Here it is the goal: with only 8 rows there is no generalization to lose, so a model that fails to overfit them is telling you the training loop is broken.

The procedure:

1. Take 8 samples.
2. Turn off shuffling, augmentation, dropout, and weight decay. The last two are regularizers, deliberate handicaps that trade training accuracy for generalization (dropout randomly switches off part of the network each step; weight decay penalizes large parameter values). Turn them off, because here you want memorization.
3. Train on those same 8 rows over and over.
4. Watch whether the loss goes to approximately zero.

The full treatment is in overfit a single batch. The reason it belongs here is the elimination argument: a model that cannot memorize 8 examples has no capacity, data-volume, or generalization explanation available. Capacity is how much structure the model can represent at all, and eight rows demand almost none; data volume cannot be the problem at eight rows; generalization cannot be the problem when nothing is being generalized to. So what remains is a short list:

• labels misaligned with inputs;
• loss applied to the wrong axis of the output array;
• frozen parameters that never receive updates;
• a missing zero_grad call, so gradients from old batches accumulate;
• lr = 0, so the optimizer takes steps of size zero.

Ten minutes of work converts an open-ended investigation into a five-item checklist.

1b. Label-noise audit

This check estimates what fraction of your stored answers are simply wrong, which caps every metric you will ever compute.

Sample 100 training rows at random and label them yourself, blind to the stored label. Then compute your agreement rate:

• Agreement 0.97+: labels are fine, move on.
• Agreement 0.88: your labels are 12% wrong, which caps everything downstream. Go to step 5 before touching the model.

Use 100 rows, not 10, because the estimate has its own standard error: sqrt(0.88*0.12/100) = 0.032, about plus or minus 3 points. That is enough to tell “clean” (0.97) from “12% noise” (0.88), and not enough to tell 8% from 12%. Do not argue about the second decimal place.

1c. Train/serve skew

Train/serve skew is a difference between the feature values computed during training and those computed at serving time for the same entity. It is the highest-yield check when offline and online numbers disagree. Compute the same feature, for the same entity (the thing a row is about, such as one user), at the same timestamp, through both code paths, and diff.

feature              offline mean   online mean   offline p50   online p50   % rows differing
purchases_7d              0.42          0.19          0.0          0.0            31%
session_len_s            184.2         181.9        142.0        141.0             4%
country_code_hash        (uniform)     (uniform)      --           --              0%
days_since_signup         88.4          88.6         61.0         61.0             1%

The column to read is the last one, the share of rows where the two paths disagree (p50 is the median). One feature is wildly off (purchases_7d differs on 31% of rows, offline mean more than double online) while the other three differ on 4%, 1%, and 0%. One bad feature against a clean background is the signature of skew. If everything were off by a little, you would be looking at a different problem, such as a sampling difference between the two log sources.

Three mechanisms produce it (feature stores and train/serve skew has the full taxonomy):

1. Different code. The offline aggregation is SQL, the online one is Python, and they disagree on < versus <= at the window boundary, on whether an event exactly at the cutoff counts.
2. Different data availability. Offline you can see the whole day; online, at 09:00, you cannot.
3. Different time semantics. Offline joins use the row’s label date; online uses “now.” This one is dangerous because it is simultaneously a skew bug and a leakage bug, the subject of 1d.

1d. Leakage

Leakage is information about the label reaching the model through its inputs. The signal is that the number is too good, not “surprisingly good” but implausibly good relative to what a domain expert achieves or what the problem could contain:

offline val AUC .................. 0.987
domain expert, same information .. ~0.75
prior production model ........... 0.81

An 0.987 on a problem humans solve at 0.75 is not a breakthrough. Two diagnostics settle it, in order: the scan finds a suspect, the ablation convicts it.

Diagnostic 1: the single-feature AUC scan. Fit each feature alone and rank the results. Any single raw feature above about 0.95 is a leak until proven otherwise, because no honest individual measurement separates the classes that cleanly.

from sklearn.metrics import roc_auc_score

def single_feature_auc(X, y, feature_names):
    """Rank features by standalone AUC. A single feature near 1.0
    is almost always leakage, not signal."""
    scores = []
    for j, name in enumerate(feature_names):
        auc = roc_auc_score(y, X[:, j])
        scores.append((name, max(auc, 1.0 - auc)))   # direction-agnostic
    return sorted(scores, key=lambda kv: -kv[1])

The max(auc, 1.0 - auc) line makes the scan indifferent to direction: a feature that predicts the answer perfectly backwards has raw AUC 0.0, and max(0.0, 1.0) = 1.0 marks it as suspicious as a perfectly-forwards feature, so one threshold catches both. Here is what the ranked output looks like. The gap between the first row and the second, not the absolute value, is what makes the top feature stand out:

feature                 standalone AUC
account_status_code          0.981      <- set to 'closed' when churn is recorded
tenure_days                  0.688
support_tickets_30d          0.640
purchases_7d                 0.612

Diagnostic 2: the confirming ablation. An ablation is retraining with something deliberately removed. Drop the suspect, retrain, and check the metric falls to a plausible level instead of merely falling. Always ablate an ordinary feature too, as a control:

all features ....................... AUC 0.987
minus account_status_code .......... AUC 0.792     <- plausible; the leak is confirmed
minus tenure_days (control) ........ AUC 0.981     <- ordinary feature, small effect

Removing the leak drops AUC to 0.792, right next to the prior model’s 0.81 and the expert’s 0.75; removing an ordinary feature drops it by 0.006. The confirmation is not “the metric dropped,” it is “the metric dropped to the range the problem actually supports.” A leak removal that leaves you at 0.96 means there is a second leak.

Four leakage patterns, each with a different mechanism and check:

A few terms: ETL (extract-transform-load) is the batch job that moves and reshapes data between systems; reading it tells you which process wrote a column. t - 1 means “as of one time unit before the decision,” so recomputing every feature at t - 1 and diffing shows which were quietly reading the future. A CV fold is one train/validation partition inside cross-validation, the practice of rotating which slice is held out. The four transforms in the last row: a scaler rescales a feature, an imputer fills missing values, a target encoder replaces a category with the average label for that category, and a feature selector keeps a subset of columns. Fit any of them on all the data before splitting and the held-out rows have influenced the transform, a quiet leak (target encoding and the exact leakage mechanism shows the worst case, where a row’s own label ends up baked into its own feature).

Group leakage is worth a number, because the intuitive estimate badly understates it. Take 40,000 rows over 8,000 users (5 rows each), split 80/20 at random by row. A user lands entirely in train with probability 0.8^5 = 0.33, entirely in test with 0.2^5 = 0.0003, so 67% of users straddle the split. Worse, pick any test row: its user has 4 other rows, and the user is absent from train only if all four also went to test (0.2^4), so 99.8% of test rows have their user in train. A per-user memorizable feature then makes essentially the entire test set memorized, and no regularization tuning will reveal it.

Step 2 — Bias or variance?

One of the most expensive planning questions in applied machine learning (would more data help?) can be measured, not guessed.

Assumption on trial: more data would help. It is roughly a coin flip whether that is true, and the two answers point at completely different quarters of work.

Two different plots get called “learning curves” and they answer different questions.

An epoch is one full pass through the training set. Everything below is about the second plot: retrain the same model on 5,000 rows, then 10,000, then 20,000, and plot validation error against the size you trained on. The chart turns the resulting shape into a diagnosis. Red is a bug (validation below train), orange is a real limitation, green is the clean outcome.

flowchart TD
    LC["Plot val error vs training-set size<br/>at 5-6 sizes, several seeds each"] --> Q1{"Train and validation<br/>curves converged?"}
    Q1 -->|"yes, at HIGH error"| B["BIAS-limited<br/>more data will not help<br/>-> capacity or features"]
    Q1 -->|"yes, at LOW error"| D["Done for this model class<br/>-> slice analysis, metric, labels"]
    Q1 -->|"no, large gap"| Q2{"Is validation error still<br/>falling with n?"}
    Q2 -->|yes| V["VARIANCE-limited<br/>extrapolate the curve<br/>to price the data"]
    Q2 -->|"no, flat"| N["Noise ceiling or<br/>distribution mismatch<br/>-> STEP 5"]
    Q1 -->|"validation BELOW train"| A["Accounting or leakage<br/>re-measure train in eval mode"]

    style B fill:#bc6c25,color:#fff
    style V fill:#bc6c25,color:#fff
    style N fill:#bc6c25,color:#fff
    style D fill:#2d6a4f,color:#fff
    style A fill:#9d0208,color:#fff

Five outcomes:

• Converged at HIGH error: bias limited. The model is too simple to represent the pattern. More data will not help; go to capacity or features.
• Converged at LOW error: you are done with this model class. Go to slice analysis, the metric, or the labels.
• Large gap, validation still falling as n grows: variance limited. Extrapolate the curve and price the data (next).
• Large gap, validation flat in n: neither. Go to step 5.
• Validation below training error: an accounting artifact or a leak. Separate them by re-measuring training error in evaluation mode, which turns off dropout.

Turning the curve into a price list

The shape of the curve turns into a dollar figure, which is the entire reason to plot it. Measured validation error at four sizes:

n        val error
 5,000     0.290
10,000     0.240
20,000     0.205
40,000     0.180

The error falls by less each doubling. Subtract a guessed floor of 0.12 and the excess over that floor multiplies by a constant 0.71 per doubling, and 0.71 = 1/sqrt(2), exactly what a quantity falling as 1/sqrt(n) does when n doubles. That is the fingerprint of a power law, so fit

err(n) = e_inf + a / sqrt(n)

read “error equals e-infinity plus a over the square root of n,” where e_inf is the asymptote the curve approaches but never crosses. Fitting gives e_inf = 0.12 and a = 12, and all four measured points reproduce. Inverting (n = (a/(e - e_inf))^2) turns the curve into a price list:

“Get more data” stops being advice and becomes a budget line: the next 3 points of error cost 4x the labeling spend, and the 2 points after that cost 36x. The e_inf = 0.12 asymptote is equally load-bearing. It is the part of the error that more of the same kind of data cannot touch, which is what sends you to features in step 6 instead of to labeling.

The five curve shapes

Each shape forbids a remedy, and the forbidden one is usually the one someone is about to propose.

The hypothesis class (first row) is the set of functions the model family can express at all; a straight line cannot bend no matter how much data it sees. The fourth row is the one that gets misread: a big gap looks like variance and gets treated with data, but if validation is flat while n quadruples, the extra data is not reaching the failure and buying four times more will not either. Go to step 5.

Step 3 — Error analysis by slice

Taking a respectable overall number apart by subgroup is where most real defects are found.

Assumption on trial: the average describes every user. It never does, and the arithmetic guarantees it: an overall metric is a traffic-weighted average, so a subgroup must be small for the average to hide its failure, and small subgroups are exactly where new markets, new devices, and new languages live.

A worked breakdown

Overall accuracy 0.922 on 20,000 rows is comfortably above baseline and gives no reason to look further. Read the bolded row against the total, and compare its share of traffic against its share of errors:

Four percent of traffic produces twenty-five percent of all errors, at an accuracy of 0.510, a coin flip. Yet if that slice performed like the rest of traffic, the headline would move by only 0.04 * (0.93 - 0.51) = 0.017, seventeen thousandths, well inside the range people casually attribute to hyperparameters. The average is arithmetically incapable of making this slice visible.

Two consequences. If mobile · es is the segment the company is expanding into, the model is unusable in the market it is being built for while the headline says it is fine. And fixing that one slice from 0.51 to 0.93 cuts its errors from 392 to 56, moving overall accuracy 0.922 → 0.939, the same 1.7 points that a quarter of model work would not buy.

Finding the cause

The slice is not intrinsically “hard.” Slice the features, not just the metric. Compare the model’s inputs inside the slice against everywhere else:

                        mobile·es    everything else
page_text non-null         22%             97%
page_text mean length      41 chars      1,180 chars
model p(class) mean       0.08            0.47

The first two rows say the model is not getting the same information: page_text is present for 97% of other traffic and only 22% here, and when present it is 41 characters instead of 1,180. The third (the model’s mean predicted probability) is 0.08 in the slice against 0.47 elsewhere: the model is confidently saying “no” to almost the entire slice.

The cause is upstream of the model. The mobile SDK (software development kit, the library the app uses to send events) truncates page_text and emits null below a length threshold; the imputer fills those nulls with 0, and the model reads 0 as a confident negative instead of “unknown.” The failure is a feature-availability difference: invisible in aggregate, total within the slice. The fix is a missingness indicator (an extra binary feature that says “this value was absent”) plus a slice-specific fallback, not a better architecture (missing values: three mechanisms, three correct answers).

Doing this systematically

Slice analysis becomes routine only when it is a function you call. The function below groups rows by whatever columns you name, computes each group’s accuracy and error count, and sorts by error share. The one subtlety it guards: slices below min_n are dropped, and dividing by the all-rows error count while dropping slices makes the shares silently fail to sum to 1. The fix is to divide by the error the report covers and report the excluded remainder separately.

import numpy as np
import pandas as pd

def slice_report(df, y_true, y_pred, by, min_n=50):
    """Per-slice accuracy AND error mass, sorted by error share.
    A slice matters in proportion to n * (1 - acc), i.e. its error count."""
    y_true, y_pred = np.asarray(y_true), np.asarray(y_pred)
    groups = [(key, g) for key, g in df.groupby(by) if len(g) >= min_n]
    reported_err = sum(int((y_true[g.index] != y_pred[g.index]).sum())
                       for _, g in groups)
    dropped_err = int((y_true != y_pred).sum()) - reported_err

    rows = []
    for key, g in groups:
        err = int((y_true[g.index] != y_pred[g.index]).sum())
        rows.append({
            "slice": key, "n": len(g), "acc": 1.0 - err / len(g), "errors": err,
            # share of the errors this report COVERS, so the column sums to 1
            "err_share": err / reported_err if reported_err else 0.0,
        })
    rows.sort(key=lambda r: -r["err_share"])
    rows.append({"slice": f"<{min_n} rows (excluded)", "n": None, "acc": None,
                 "errors": dropped_err, "err_share": None})
    return rows

Sort by error share, not accuracy: a 30-row slice with two errors has accuracy 0.933 but contributes 2 of 1,560 errors, so accuracy floats it near the top of your worry list while error share ranks it by how much fixing it would actually move the headline. Three practices make this routine:

1. Slice on everything cheap: device, locale, tenure bucket, traffic source, time of day, input-length decile, and the label itself (which gives per-class recall). Then slice on pairs of the top offenders.
2. Automate the search. Fit a shallow decision tree whose target is the error indicator (1 when the model got the row wrong, else 0). Its high-error leaves with non-trivial support are your slices, discovered, not guessed.
3. Report worst-slice metrics as a standing number. A model whose mean rose and whose worst slice fell is usually a regression.

Step 4 — Is the metric wrong?

Sometimes the model behaves correctly and the number describing it is wrong, in which case fixing the model would not have helped.

Assumption on trial: the metric goes up exactly when the product gets better. Five distinct mechanisms break that link (imbalance, the wrong unit of aggregation, an inherited threshold, missing calibration, and proxy drift) and each has a different repair.

Accuracy and AUC under imbalance

You saw one form in step 0, where 0.940 accuracy loses to a constant at 0.942. The subtler version is ROC-AUC. With 100,000 negatives and 1,000 positives (100:1), suppose your operating point (the threshold at which scores become decisions) gives recall 0.80 at a false-positive rate (FPR) of 0.05. Count what lands on an analyst’s desk:

TP = 0.80 * 1,000   =    800     <- positives caught
FP = 0.05 * 100,000 =  5,000     <- negatives flagged by mistake
precision = 800 / 5,800 = 0.138

Precision of 0.138 means 6.25 false alarms for every real one, while the FPR of 0.05 looks respectable. The mechanism is normalization: ROC-AUC’s x-axis is the false-positive rate, which divides by the negative count, so a 100:1 imbalance cancels out of it entirely. Precision does not divide by the negative count, which is why PR-AUC (area under the precision-recall curve) and precision-at-fixed-recall show you this and ROC-AUC does not. Report both: ROC-AUC is stable across prevalence, so it is the one you can compare across datasets or time; PR-AUC is meaningless unless you state its prevalence, but it is the one that shows the workload at your operating point (PR-AUC vs ROC-AUC under heavy imbalance).

The aggregation unit is wrong

A metric can be correct per row and meaningless per user. Per-row accuracy 0.970 sounds strong, but if the product only counts a session as successful when all 20 of its rows are right, and errors are independent, the session succeeds with probability 0.97^20 = 0.544. Only 54% of sessions come out fully correct while the model is 97% accurate, and both numbers are true. Report the metric at the unit the user experiences (per session, per document, per invoice), not per token or per row.

The threshold is inherited, not chosen

0.5 is a default, not a decision. Say a false negative (a miss) costs 20x what a false positive costs, so C_FN = 20 * C_FP. You should predict positive when the expected cost of silence exceeds the expected cost of an alarm; setting them equal gives the break-even threshold

p* = C_FP / (C_FP + C_FN) = 1 / 21 = 0.048

read “p-star.” The cost-optimal threshold is about 0.048, not 0.5, so every row scored between 0.048 and 0.5 should be alarmed and, on the default, is not.

def cost_optimal_threshold(c_fp: float, c_fn: float) -> float:
    """Predict positive when expected cost of a miss exceeds a false alarm:
    p*C_fn > (1-p)*C_fp  ->  p > c_fp / (c_fp + c_fn)"""
    return c_fp / (c_fp + c_fn)

The trap this produces (comparing two models at an inherited threshold) is worked through in mini-case D and derived at length in choosing a threshold from the cost matrix.

Calibration, when a probability is consumed as a number

Calibration means that among rows the model scores 0.30, about 30% really are positive. Whether it matters depends on what happens to the score after the model emits it:

• The score is only thresholded (flag / do not flag). Then only its ranking matters, and AUC is the right metric.
• The score is multiplied by something: expected revenue, expected loss, a bid. Then calibration is the metric, and AUC cannot see it.

AUC cannot see it because any monotone transform of the scores (any relabelling that preserves their order, such as squaring every score) leaves AUC unchanged. AUC only asks which of two scores is larger. Squaring turns 0.30 into 0.09, destroying calibration while AUC reports no change. To measure it, build a reliability table: bucket rows by predicted score and compare each bucket’s mean prediction against the fraction that actually turned out positive.

bucket        n      mean predicted p     observed rate
0.0 - 0.1   4,120        0.041               0.038
0.1 - 0.3   2,880        0.198               0.212
0.3 - 0.6   1,540        0.442               0.590        <- underconfident here
0.6 - 0.9     980        0.744               0.881
0.9 - 1.0     480        0.951               0.972

ECE = SUM (n_b/n) * |pred_b - obs_b| = 0.042

ECE is the expected calibration error: the average gap between predicted and observed rate, weighted by how many rows each bucket holds. An ECE of 0.042 means an expected-value calculation built on these scores is off by four points of probability on average, and every AUC-based report will say the model is fine. Two standard fixes, both fitted on a held-out calibration split: isotonic regression fits a monotone step function from raw score to calibrated probability, and Platt scaling fits a one-dimensional logistic curve instead, which is smoother and needs less data. Re-fit on every deploy. Calibration is the first thing distribution shift breaks (Platt scaling vs isotonic regression).

The metric is a proxy for something else

The last case no diagnostic catches, because the metric is computed correctly and still points the wrong way. Every offline metric stands in for something the business wants, and optimizing hard against the stand-in eventually produces the stand-in without the thing:

• Click-through rate stands in for “the user found this useful.” Optimize it and you get clickbait.
• Session length stands in for engagement. Optimize it and you get confusion.
• Per-token accuracy stands in for a good answer. Optimize it and you get hedging.

When the offline metric improves and the business metric does not, the burden of proof is on the offline metric. A proxy that has been optimized against for two quarters is no longer a proxy. That is Goodhart’s law, the same mechanism as reward hacking (RLHF). See also why offline ranking metrics disagree with online CTR.

Step 5 — Is the label wrong?

Stored answers are a measurement with their own error rate, and that error rate imposes a ceiling.

Assumption on trial: the test label is the truth. When it is not, the test set stops being a ruler and becomes a second noisy model that yours is scored against, and above a certain point the scoring can rank a better model below a worse one.

The noise ceiling

Two symbols carry the derivation: a is the model’s accuracy against the true label (unobservable, which is the point), and eta (η) is the label error rate, the probability that a stored label was flipped away from the truth. You score against the stored label, so the model gets credit two ways, right prediction and unflipped label, or wrong prediction and a label flipped to match it:

observed = a*(1 - eta) + (1 - a)*eta            (binary, C = 2)
invert:    a = (observed - eta) / (1 - 2*eta)

Set a = 1 (a perfect model) and the second term vanishes: a perfect model cannot score above 1 - eta. That is the ceiling.

With 12% label noise, a measured 0.880 is what a perfect model scores, and a measured 0.850 corresponds to a true accuracy of 0.961. Your model is far better than the report says, and no further training will move the reported number. The last row is a useful consistency check: at eta = 0.20 a perfect model tops out at 0.800, so an observed 0.85 is impossible. An observed score above 1 - eta is proof that your noise estimate is wrong, your test set is leaking, or the noise is not symmetric.

The inversion is binary-only. Reason 2 above (“wrong prediction and label flipped to match it”) assumes a wrong prediction and a flipped label always coincide, which is true only with two classes. With C classes a flip matches a wrong prediction only 1/(C-1) of the time:

observed = a*(1 - eta) + (1 - a) * eta/(C - 1)
invert:    a = (observed - eta/(C-1)) / (1 - eta - eta/(C-1))

The ceiling 1 - eta is unchanged for every C (set a = 1 and the second term still vanishes), so the “max observed” column survives but the “true accuracy implied” column does not. Read that column as binary only.

Estimating eta

1. Draw a random sample of 200 test rows.
2. Have three independent annotators label them, blind to the stored label and to each other.
3. Take the majority as the reference; eta is the disagreement rate against the stored label.
4. Report inter-annotator agreement as Cohen’s kappa between annotator pairs.

Step 4 is the one people skip, and the one people misuse: a kappa is not an accuracy, and quoting it directly as a ceiling understates the ceiling badly. Two calculations are needed.

Cohen’s kappa (κ) is chance-corrected: it subtracts off the agreement two annotators would reach by guessing (p_e, the chance agreement given class frequencies) and rescales so 1.0 is perfect and 0.0 is chance. Undo that correction first to recover the raw agreement p_o:

p_o = kappa*(1 - p_e) + p_e

kappa = 0.62, raw agreement p_o by task shape:
   balanced binary    p_e = 0.50  ->  p_o = 0.810
   balanced 3-class   p_e = 0.333 ->  p_o = 0.747
   90 / 10 binary     p_e = 0.820 ->  p_o = 0.932

The conversion depends on class balance, so there is no single number to memorize. The same kappa of 0.62 means 0.81 agreement on one task and 0.93 on another. Then turn agreement into a ceiling. If two annotators each have error rate eta, they agree when both are right or both wrong the same way: (1-eta)^2 + eta^2 = p_o. Substituting the balanced-binary p_o = 0.81 and solving gives eta = 0.106, so the ceiling is 1 - eta = 0.89. The ceiling is 0.89, not 0.81. The annotators disagree with each other more than either disagrees with the truth, because two independent 10.6% error rates compound. A model reported above 0.89 is fitting one annotator’s idiosyncrasies and will not transfer.

Finding the mislabeled rows cheaply

Let the model nominate the suspects: the rows it is most confidently “wrong” about are disproportionately rows where the model is right and the stored label is wrong. Rank the test set by per-row loss and read the top 100 by hand. The confident-learning version formalizes this: get out-of-fold predicted probabilities (predictions for each row from a model that never saw it in training, so they are honest) and flag rows where the model assigns high probability to a class other than the stored one.

rank   stored label   p(stored)   p(argmax)   argmax class   verdict on review
  1    "billing"        0.002       0.981      "shipping"     label wrong
  2    "billing"        0.004       0.955      "shipping"     label wrong
  3    "technical"      0.006       0.943      "billing"      genuinely ambiguous
  4    "shipping"       0.008       0.902      "billing"      label wrong
...
      of the top 100:  38 label errors, 22 ambiguous, 40 real model errors

p(stored) is the probability the model assigned to the label on file; p(argmax) is the probability of its own top choice. Here only 40 of the 100 worst-scoring rows are actually model errors. The other 60 are problems with the test set. The exact fraction is a property of your dataset, not a constant, so measure it instead of quoting one; published audits of standard benchmarks land in a similar region.

Train noise and test noise are not the same problem

Clean the test set first. The cost ratio is just the size ratio: on the running example, 200,000 training rows against 2,000 test rows makes relabelling the test set 100x cheaper at the same price per row (against a 20,000-row test set it is only 10x, so state both sizes). The second reason does not depend on size, since the test set is what you steer by, so until it is clean you cannot tell whether cleaning the training set helped.

Step 6 — Capacity, then features, then data

This is the branch where you improve the model, reached only when no assumption turned out to be broken, so the model really is the bottleneck. What remains is a budgeting question, and the order is set by information gained per hour, not preference. The fills below encode cost only.

flowchart LR
    C["1. CAPACITY<br/>hours · no new data<br/>DIAGNOSTIC as well as a fix"] --> F["2. FEATURES<br/>days-weeks · adds information<br/>needs offline/online parity"]
    F --> D["3. DATA<br/>weeks-quarters · price known<br/>from the STEP 2 curve"]

    style C fill:#2d6a4f,color:#fff
    style F fill:#40916c,color:#fff
    style D fill:#bc6c25,color:#fff

1. Capacity first, because it is also a diagnostic. Capacity is roughly how many parameters the model has and how flexibly they combine. Double the width or depth, remove regularization, and watch training error. You are asking what the model is able to fit, not what it generalizes to.

config             train err   val err     conclusion
base                 0.171       0.180     --
2x width             0.166       0.179     not capacity-limited: train barely moved
2x width, no wd      0.088       0.181     capacity was there; regularization was binding
4x width, no wd      0.021       0.186     now variance-limited -> data or regularization

wd is weight decay. Read the rows in order: 2x width barely moves training error (the model was not straining against its size); removing weight decay drops it to 0.088 (the capacity was there, regularization held it back); 4x width without weight decay drops training error to 0.021 while validation rises (variance). Reading this table top to bottom is the bias-variance diagnosis, and it costs an afternoon. The decisive case this example does not show: training error that will not fall even with the handicap removed means the model is not the limitation. You have a data-representation problem, and going straight to data collection would have burned a quarter to learn the same thing.

2. Features second, because they are the only lever that adds information. Capacity re-uses the information already in the inputs; a feature brings information that was not there. When step 2’s curve shows a high e_inf asymptote, that asymptote is precisely the part more data cannot reach and a new feature can. The cost is real: every feature needs a point-in-time-correct offline computation (one that can only see what was knowable at the decision moment), an online implementation, and a parity test between them, which is why the skew check in step 1c is a permanent tax on this step.

3. Data last, because it is the slowest and the one whose value you can already price. Step 2’s fitted curve already said it: 4x the data for 3 points of error, 36x for 5. Bringing a number like that to a planning meeting is a different conversation from “we need more data.”

The one legitimate reorder: a starved slice jumps the queue. If step 3 found one, targeted collection for that slice goes first, because the two purchases are not comparable: 5,000 mobile · es rows is a week of work worth 1.7 points of overall accuracy, while 5,000 more rows of the dominant slice is worth roughly nothing (that slice is already far out on the flat part of its own curve).

Symptom to broken assumption to diagnostic to fix

This table is the playbook in lookup form: find your symptom, read across. Read the second column first. Naming the assumption turns a symptom into a search with an end, because an assumption can be tested and a symptom cannot.

A few shorthands: ln(C) is the natural log of the number of classes, exactly the loss of a model outputting a uniform guess (ln(10) = 2.30 for ten classes). A loss pinned there means the model is guessing. requires_grad is the PyTorch flag deciding whether a parameter accumulates gradients at all; set to False, it never moves. eval() switches dropout and other train-only behaviour off; measure training loss without it and the loss is inflated. EV is expected value (the score multiplied by an amount of money). PSI is the population stability index, which compares this week’s feature distribution against the training window’s and grows as they diverge (PSI and KL computed). An A/B test splits live traffic between two variants and compares outcomes.

The last row’s three remedies all make the rare class count for more: class weights multiply its contribution to the loss, resampling changes how often its rows appear, and focal loss down-weights examples the model already gets right so the rare, hard ones dominate the gradient (focal loss). Its two diagnoses: a collapsed head is a final layer emitting one class regardless of input, and a degenerate optimum is the situation that rewards it. Under enough imbalance, always guessing the majority genuinely minimizes the loss you wrote down, so the model is fine and the objective is broken.

Worked mini-cases

Each case follows the same shape: signal, first diagnostic, the assumption that turned out false, the mechanism, the fix, and a verification that would have failed if the diagnosis were wrong. Each is chosen because its correct resolution looks wrong at first glance.

A. “AUC 0.94 offline, 0.71 online”

Signal. Offline validation AUC is 0.940 on a time-based split. Two weeks live, online AUC is 0.712 on the same population.

Broken assumption. The feature computed during training is the same quantity as the feature computed at serving time.

First diagnostic. Replay one day of serving logs through the offline scorer and diff the feature vectors per entity. Do not touch the model.

feature            offline mean   online mean   importance share
purchases_7d            0.42          0.19            0.61          <- 61% of the model
tenure_days            88.4          88.6            0.11
support_tickets_30d     0.31          0.30            0.08

The third column is importance share: for a tree ensemble, a feature’s gain is the total loss reduction from every split made on it, and its importance share is that gain divided by the total over all features (so the column sums to 1). A share of 0.61 means this one feature accounts for 61% of the model’s loss reduction, and it is the one whose offline mean (0.42) is more than double its online mean (0.19). (Importance share says “how much of the model this feature is,” not “how much accuracy you would lose by dropping it”; the second is what the ablation in step 1d measures.)

Mechanism. The purchases_7d aggregation was computed offline with BETWEEN label_date - 7 AND label_date, including purchases made on the label day, after the event the model is supposed to predict. Online, the window can only reach t - 1. This is a lookahead leak and a skew bug at once: the model’s dominant feature was partly a copy of the label offline, and a genuinely weaker feature online.

Fix. Use point-in-time-correct (as-of) joins so the offline feature can only see what was visible at the decision timestamp, then move to one shared implementation for both paths.

Verification. Read the gap:

                 before      after
offline AUC       0.940      0.791
online AUC        0.712      0.784
gap               0.228      0.007

Offline AUC fell by 0.149; online rose by 0.072; the gap collapsed from 0.228 to 0.007. The correct outcome of fixing leakage is that your offline number gets worse. A team that treats the offline drop as a regression will revert the fix and re-break production. Verify on the gap, never on the offline metric alone.

B. “Accuracy 0.922, and the model is unusable”

Signal. Aggregate accuracy sits comfortably above baseline, while support tickets concentrate in one market.

Broken assumption. The overall accuracy describes what any given user experiences.

First diagnostic. Run slice_report on (device, locale), sorted by error share, the table in step 3. mobile · es is 4.0% of traffic, 25.1% of errors, and 0.510 accuracy.

Mechanism. page_text is non-null for 97% of other traffic and 22% of this slice, because the mobile SDK drops it below a length threshold; the imputer fills nulls with 0, which the model cannot distinguish from a genuine zero, so it emits a confident low score for almost the entire slice (mean p = 0.08 versus 0.47 elsewhere).

Fix, in order. Add a missingness indicator so “absent” and “zero” are different inputs; add a slice-appropriate fallback feature the mobile SDK does emit; then collect 5,000 labeled rows from that slice.

Verification. Slice accuracy goes 0.510 → 0.930 and overall goes 0.922 → 0.939; the worst-slice metric then goes on the standing dashboard, so the next occurrence is caught by an alarm instead of by tickets.

C. “The model will not learn at all”

Signal. Training loss is pinned at 2.303 for 4,000 steps on a 10-class problem. Since ln(10) = 2.30, the model is outputting a uniform distribution over the ten classes and never leaves it.

Broken assumption. Each input is paired with its own label.

First diagnostic. Overfit 8 samples. It fails. The loss stays at 2.3025 for 400 steps on a batch a linear model could memorize:

healthy (8 samples, 10 classes)     this run
   0   2.3026                          0   2.3026
  50   1.9034                         50   2.3025
 200   0.1107                        200   2.3026
 400   0.0009                        400   2.3025

Mechanism. Once 8 samples cannot be memorized, learning rate, initialization, and capacity are excluded by construction, so the answer is in the data. Printing one batch finds it: the dataset builder shuffled the image list and the label list with two separate random.shuffle calls, so each image ended up next to some other image’s label. Every label was valid, every image was valid, and the pairing was uniformly random, for which the best possible prediction (the Bayes-optimal output) is exactly the uniform distribution the model settled on. The model was right; the data was a permutation.

Fix. Shuffle indices, not lists, so images and labels move together. Add a permanent assertion that the loaded pair’s filename stem matches the label record’s key.

Verification. The 8-sample test reaches a loss below 1e-3 in 400 steps and full training reaches 0.34 validation loss. The single-batch test then becomes a continuous-integration gate. It runs automatically on every commit, because this class of bug costs days and is caught in ten minutes.

D. “F1 went from 0.62 to 0.68 and alerts dropped 38%”

Signal. The offline report says the new model is better on F1. After deploy, alerting volume fell by 38% and analysts started finding missed cases.

Broken assumption. A threshold tuned for one model means the same thing for another.

First diagnostic. Evaluate both models at the deployed threshold, not at each model’s own tuned threshold. The second and third rows are the same model:

                       threshold   precision   recall    F1
model A (deployed)        0.50       0.55       0.71     0.620
model B (offline report)  0.28       0.62       0.76     0.683   <- what the report measured
model B (as deployed)     0.50       0.83       0.44     0.575   <- what production got

At 0.28, B beats A on F1 (0.683 vs 0.620). At 0.50, the same B loses to A (0.575 vs 0.620) because its recall collapses from 0.76 to 0.44. That recall collapse is the 38% drop in alert volume.

Mechanism. Model B is genuinely better: it dominates A on the precision-recall curve. But B was trained with different regularization and its score distribution is shifted left, so the same numeric threshold sits at a different operating point on B than on A. The offline report tuned B’s threshold to 0.28; the deploy inherited A’s 0.50.

Fix. Ship the threshold as part of the model artifact, chosen on a held-out set by the actual cost ratio (step 4), never as a constant in the serving code. Report threshold-free curves such as PR-AUC and the metric at the exact operating point you will deploy.

Verification. Re-deploy B at 0.28: recall goes 0.44 → 0.76, alert volume returns above A’s, and precision stays better than A’s 0.55. Then add a deploy-time check that fails if the new model’s predicted-positive rate at the configured threshold differs from the incumbent’s by more than a set tolerance. That single guard catches this entire class of failure.

Quick reference

One row per idea, with the mechanism or number that makes it load-bearing.

Conclusion

The playbook is a fixed order, and the order is what makes it work. Reproduce and baseline before you trust any number; separate a data bug from a model bug before you touch the model; price data with a learning curve before you buy it; and only change the model once leakage, skew, slices, the metric, and the labels have all been ruled out. Most underperformance in applied work is a data, label, metric, or skew problem. The model is the residual. Every step ends in the same sentence: a named broken assumption, the number that proves it, the fix, and the check that would have failed if you were wrong.

One line to remember: a bug is an assumption you did not know you were making, so debug the ladder in order and stop at the first step whose number proves an assumption false.

Further reading

• Andrew Ng, Machine Learning Yearning. Error analysis, bias/variance, and how to prioritize what to fix.
• Northcutt, Jiang, and Chuang, “Confident Learning: Estimating Uncertainty in Dataset Labels” (2021). The out-of-fold method for finding mislabeled rows.
• Northcutt, Athalye, and Mueller, “Pervasive Label Errors in Test Sets Destabilize ML Benchmarks” (2021). Measured label-noise rates in standard benchmarks and how they reorder model rankings.
• Sculley et al., “Hidden Technical Debt in Machine Learning Systems” (NeurIPS 2015). Train/serve skew, feedback loops, and pipeline fragility in production.

Next: the math track, starting with 01 — Probability, the estimators, intervals, and hypothesis tests that every number in this chapter relies on.