Imbalanced Data, Calibration, and Drift

In this lesson, we’ll work through three problems that look separate but share one cause:

• what to do when the class you care about is rare,
• what to do when the probabilities your model reports are wrong,
• what to do when the live data stops resembling the training data.

We’ll tell them apart, derive the one-line correction that fixes two of them exactly, and pin down the assumption that correction rests on, so you can recognize when it stops holding. By the end you’ll be able to name which of a classifier’s two outputs a system consumes, apply the prior-shift correction, and read the drift signal that arrives before the labels do.

Everything here concerns a classifier: a model that takes a row of features x (one transaction, one loan application, one image) and returns p(y=1|x), the probability that the row belongs to the class you care about, a number between 0 and 1.

The label y is 1 for the class of interest (fraud, click, disease) and 0 otherwise. By convention y = 1 is the positive class and y = 0 the negative class.

That single output feeds two different kinds of consumer:

• Ranking is the order the scores put the rows in. A top-k queue, a search page, or a fixed-capacity review list uses nothing but the order.
• Probability is the number taken at face value. An expected-loss calculation, an insurance price, or a rule like “auto-block above 0.99” reads the number itself.

Which of the two your system consumes is the organizing question of the chapter. The two break for different reasons and are repaired by different tools.

All three topics of the title are the same thing underneath: a prior that moved. The prior is p(y=1), the fraction of all rows that are positive before you look at any features, the base rate.

• Imbalance is a prior that is extreme.
• Resampling (changing the mix of positives and negatives in training on purpose) is a prior you moved deliberately.
• Label shift is a prior that moved on its own, with no change you made.

In all three the repair is the same correction in log-odds space. The log-odds of a probability p is ln(p/(1-p)): the scale on which a change of prior stops being a messy rescaling and becomes plain addition. And in all three the failure looks like “the model got worse” when nothing about p(x|y) (the distribution of feature values within a class) has actually changed.

So a classifier’s ranking and its probabilities are separate objects, damaged by separate things, repaired by separate tools. The first question in every section is whether anything downstream reads the number, or only the order.

Imbalance is usually a threshold problem, not a data problem

Class imbalance means one class is far more common than the other: 1% fraud against 99% legitimate is a 1:99 imbalance. The reflex on hearing “1% positives” is to rebalance the data, but the first move is to ask what actually broke, since in most cases the model is fine and something downstream of it is not.

The triage

Ask two questions in order. First, how many positive rows you have in absolute terms: that separates a genuine shortage of data from a mere ratio. Second, if you have plenty of positives, which specific downstream component is misbehaving.

flowchart TD
    S["1% positive class"] --> Q1{"How many positives<br/>in absolute terms?"}
    Q1 -->|"under ~1,000"| DATA["A real DATA problem.<br/>The model cannot estimate<br/>the minority conditional at all.<br/>Get labels, add features,<br/>use a simpler model."]
    Q1 -->|"thousands or more"| Q2{"What is broken?"}
    Q2 --> M1["Metric is accuracy<br/>-> switch to a metric that<br/>is not majority-dominated"]
    Q2 --> M2["Decisions use argmax<br/>at p = 0.5<br/>-> derive threshold from cost"]
    Q2 --> M3["Optimizer is drowned by<br/>easy negatives<br/>-> class weights or focal loss"]
    Q2 --> M4["Nothing. Log loss already<br/>fits a low-prior model fine.<br/>-> do nothing"]

    style DATA fill:#9d0208,color:#fff
    style M2 fill:#2d6a4f,color:#fff
    style M4 fill:#1d3557,color:#fff

The second question has four answers:

• The metric is accuracy. Switch to a metric that is not dominated by the majority class, as in the confusion matrix section of the metrics chapter.
• Decisions use argmax at p = 0.5. Argmax takes whichever class has the larger score, which for two classes is exactly thresholding the probability at 0.5. Derive the threshold from the cost of each kind of mistake instead, as in choosing a threshold from the cost matrix.
• The optimizer is drowned by easy negatives. The training signal is dominated by the enormous number of obviously-negative rows. Fix it with class weights or focal loss, both below.
• Nothing is broken. Log loss already fits a low-prior model perfectly well. Do nothing.

The useful first move on a 1:99 problem is to check whether there is an imbalance problem at all: count absolute positives, name the metric, name the decision threshold and where it came from. Reaching straight for SMOTE skips that check.

Relative rarity and absolute rarity are different problems

1% of 10,000,000 rows is 100,000 positives, more minority examples than most datasets have rows, and nothing is wrong. 1% of 2,000 rows is 20 positives, and no resampling technique creates information that 20 examples do not contain.

SMOTE, the Synthetic Minority Over-sampling Technique, manufactures new minority rows by interpolating between existing ones (dissected in Resampling and the calibration it breaks). Run on 20 points it produces convex combinations of 20 points (points sitting somewhere on the straight line between two of them) and no new information.

Where the diagram’s ~1,000 boundary comes from

It is not a measured constant. It is roughly where the minority conditional p(x|y=1) (the distribution of feature values among the positive rows) acquires enough support (enough observed examples, spread over enough of the feature space) to estimate anything. Two independent readings land on the same order of magnitude.

Events per variable. Logistic regression, the standard linear model for two-class problems, fits one coefficient per feature, and a rule of thumb asks for 10–20 positive events per coefficient. A 50-feature model therefore wants 500–1,000 positives.

Metric noise. Sampling noise on any minority-class metric scales as 1/sqrt(n_pos). For a recall of 0.60 the 95% error bar is about ±22 points at 20 positives, ±3 points at 1,000, and ±0.3 points at 100,000. Below roughly a thousand positives the error bar is wider than the range of recalls you would argue about: you cannot measure a change you make, never mind learn one.

Read ~1,000 as an order of magnitude that moves with your feature count and your metric resolution, not a test.

The model is fine; what breaks is downstream

A 1% positive rate does not damage the model. Train a logistic regression or a GBDT (a gradient-boosted decision tree, an ensemble that adds up many small trees, each fitted to the errors of the ones before it) with plain log loss and it converges to a good estimate of p(y|x): correct, and simply small nearly everywhere.

Log loss (cross-entropy) charges -ln(p) when the true label is positive and -ln(1-p) when it is negative, so it punishes confident mistakes savagely and is minimized by telling the truth. It is a proper scoring rule: the prediction that minimizes it in expectation is the true probability itself, whatever the prior (proper scoring rules).

So the model is fine and what breaks is downstream. The five complaints filed under “the imbalance problem,” and what each actually is:

Terms used there:

• Prevalence is the fraction of rows that are positive (1% here). A prevalence-weighted metric depends on that fraction, not only on the model.
• PR-AUC is the area under the precision-recall curve. It ignores true negatives, so it does not flatter a model for dismissing 99% of the data.
• MCC, the Matthews correlation coefficient, is one number in [-1, +1] built from all four confusion-matrix cells; it is only high when the model does well on both classes.
• Expected cost assigns a dollar figure to each mistake (C_fp for a false positive, C_fn for a false negative) and sums what the model’s errors bill you. The optimal threshold C_fp/(C_fp+C_fn) is the cutoff at which one more false positive costs exactly what one more false negative does.
• Anomaly detection, in the last row, gives up on learning the positive class from examples: you model what “normal” looks like and flag whatever falls far outside it. That is the honest move when 20 positives is all you will ever have.

Resampling, and the calibration it breaks

Resampling alters the composition of the training set on purpose (duplicating minority rows, discarding majority rows, or inventing synthetic minority rows) so the two classes appear in a ratio you chose, not the one nature supplied. The consequence, derived below: resampling leaves the model’s ranking untouched and shifts its probabilities by a fixed, computable amount.

Calibration is the property that breaks. A model is calibrated when its stated probabilities match reality, so that among the rows it labels “70% likely,” about 70% really are positive (defined properly later). Here you only need to know that resampling breaks it, and by exactly how much.

The argument in one picture

You train on a rebalanced set whose prior is pi' instead of the true pi. All you changed was how many rows of each class you kept, so p(x|y) is unchanged and only the prior moved. That multiplies the posterior odds p/(1-p) by a constant c. Taking logs turns the multiplication into logit(p) = logit(p') + ln(c), a pure intercept shift, where the logit is the log-odds ln(p/(1-p)).

That shift splits two ways. AUC is unchanged, because a constant shift is a monotone map (one that stretches the numbers without reordering them) and AUC reads only order. But every probability is wrong: a printed 0.90 can really mean 0.083. The repair is to add ln(c) to every logit; it is exact and costs one constant.

The exception is SMOTE, which changes p(x|y=1) itself, so no single constant undoes it and you must fit a calibrator instead: a small model, fitted on held-out data, that maps the raw score to an honest probability.

flowchart LR
    A["Train on a rebalanced set<br/>prior pi' instead of pi"] --> B["p of x given y unchanged<br/>only the prior moved"]
    B --> C["posterior ODDS multiplied<br/>by a constant c"]
    C --> D["logit p = logit p' + ln c<br/>a pure INTERCEPT shift"]
    D --> E["AUC unchanged<br/>monotone map, ranks preserved"]
    D --> F["Every probability wrong<br/>0.90 really means 0.083"]
    F --> G["Add ln c to every logit<br/>-- exact, one constant"]
    B -.->|"SMOTE breaks<br/>this assumption"| H["p of x given y=1 CHANGED<br/>-> no constant undoes it<br/>-> must fit a calibrator"]

    style D fill:#1d3557,color:#fff
    style E fill:#2d6a4f,color:#fff
    style F fill:#bc6c25,color:#fff
    style H fill:#9d0208,color:#fff

AUC is the area under the ROC curve, and it has an exact meaning: the probability that a randomly chosen positive row scores higher than a randomly chosen negative one (derivation). It knows nothing about the face value of the scores.

The three moves, the one non-move, and what each distorts

People do four things about imbalance. The first three change the data; the fourth changes only the loss function.

Vocabulary: the minority class is the rarer one (the positives, here); the majority class is the common one. A high-variance learner is a model flexible enough to fit the training rows almost exactly: a decision tree can isolate a row it saw 40 times into its own leaf (a terminal node holding one prediction) and score it perfectly, so training loss falls while nothing transfers. w_c multiplies the loss of every row in class c, so the optimizer treats one rare row as if it were w_c rows.

Each technique rests on an assumption. Oversampling assumes duplicated rows carry the same information as fresh ones, false for anything that can memorize. Undersampling assumes the discarded negatives were redundant, false the moment the negative class has structure, when rare-but-legitimate patterns come back as false positives. Class weights assume the only problem is the relative emphasis of the two classes, exactly right, which is why they distort nothing structural.

SMOTE assumes the segment between two minority points is minority territory, and there are three standard ways that is false:

1. A non-convex or multi-modal minority class. Multi-modal means the positives form two or more separate clusters; non-convex means the straight line between two positives can leave positive territory. Card-testing fraud lives at small amounts and account-takeover fraud at large ones; interpolating between the clusters manufactures “medium-amount fraud” that does not exist, and false positives rise in the region SMOTE invented.
2. Categorical features. A categorical feature takes one of a fixed set of unordered values (a country code), usually one-hot encoded as one 0/1 column per value. Interpolating those columns produces country_US = 0.5, which is no country at all, off the data manifold, the thin region where real rows live. SMOTE-NC substitutes the most frequent neighbour category instead, which erases the rare-category structure you were trying to model.
3. High dimensions. As feature count p grows, all nearest neighbours become nearly equidistant (the curse of dimensionality), so “nearest minority neighbour” degenerates toward “random minority point.” SMOTE then contracts the minority cloud toward its centroid and reduces recall on the tails you cared about.

The fourth failure is procedural. Cross-validation (CV) estimates out-of-sample performance by splitting rows into folds, training on all but one, and scoring the held-out fold. A synthetic point is a convex combination of two real minority points, so oversampling before the split leaves a validation row partially present in training, a leak, evaluation data influencing the fitted model:

                             CV AUC    holdout AUC
SMOTE applied, then split      0.987       0.713
split, then SMOTE in-fold      0.844       0.826
class weights, no resampling   0.851       0.839

The first row’s 0.987 is memory, not generalization. Any resampling, weighting, or calibration step belongs strictly inside the cross-validation fold.

The calibration break, derived

The one assumption: resampling changes the class prior from pi to pi' while leaving the class-conditional densities f_1(x) and f_0(x) untouched. f_1 describes what fraudulent transactions look like, f_0 what legitimate ones look like; keeping or dropping whole rows at random changes how many of each you have without changing what either looks like. That holds for random over/undersampling and class weights, and is emphatically not true for SMOTE.

Under that assumption, write the posterior odds under each prior and divide:

p(y=1|x)  =  pi·f_1(x) / (pi·f_1(x) + (1-pi)·f_0(x))          Bayes' rule

odds:      o  = p/(1-p)   = [ pi  / (1-pi ) ] · f_1(x)/f_0(x)     true world
           o' = p'/(1-p') = [ pi' / (1-pi') ] · f_1(x)/f_0(x)     resampled world

divide:    o  = o' · [ pi/(1-pi) ] · [ (1-pi')/pi' ]  =  o' · c

Converting to odds cancels the denominator; what survives is the likelihood ratio f_1(x)/f_0(x), which is identical in both worlds because resampling did not touch f_1 or f_0. Dividing the two worlds’ odds cancels it entirely, leaving only prior terms. That leftover c does not depend on x: it is the same number for every row, which is why one constant can undo the damage. Taking logs:

logit(p) = logit(p') + ln(c)

Resampling shifts the log-odds by a constant. It is an intercept bug, not a model bug. Applying the correction leaves AUC unchanged (a constant shift preserves ranks) while fixing every probability. Resampling itself is a different fit on less data, so its AUC does move (that is what the CV trace above shows), but applying the correction moves no rank.

Worked: a model trained on a 50/50 balanced set

The commonest case. True prevalence pi = 0.01, rebalanced to pi' = 0.5, so c = (0.01/0.99)·(0.5/0.5) = 0.010101 and ln c = -4.5951. Convert each output to odds, multiply by c, convert back:

A model trained on balanced data that says “90% confident” means 8.3%. Feed that 0.90 into an expected-loss calculation or a 0.5 threshold and every downstream number is wrong by an order of magnitude, and the break-even row shows the balanced model must be 99% sure before the real-world posterior even reaches a coin flip.

Worked: negative downsampling, the case you will actually meet

In negative downsampling you keep every positive row and keep each negative with probability w; w = 0.05 retains one negative in twenty. Every large-scale system does this, for compute and not statistics: keeping all the negatives would mean training on tens of billions of rows for no statistical gain. Substituting pi' = pi / (pi + w(1-pi)) into c, the prevalence cancels:

pi = 0.01,  w = 0.05
pi' = 0.01 / 0.0595 = 0.168      rows kept = 0.0595 N  ->  16.8x smaller training set
c   = w = 0.05          ln(c) = ln(0.05) = -2.9957

c collapses to w itself, so the offset is ln(w) at every prevalence. You do not need to know the true prevalence to fix a downsampled model, only the sampling rate you chose, which you always know. A downsampled model emitting p' = 0.95 (odds 19.0) corrects to odds 19.0 × 0.05 = 0.95, so p = 0.487, half what it said.

The ad-CTR chapter runs this same w = 0.05 and -2.9957 through an ad auction, pricing the missing offset at a 1.95x inflation of eCPM (effective cost per thousand impressions). It also gives the Fisher-information argument for why 16.8x less data costs only 9% in standard errors: the discarded rows were the least informative ones.

Extending from two classes to K classes

The derivation used only that the class-conditional densities did not move, which holds for any number of classes. Apply the same odds argument to each class against a common reference and you get a per-class offset on the logits, the raw pre-softmax scores z_k, where softmax turns raw scores into probabilities that sum to one:

z_k_corrected  =  z_k + ln( pi_new_k / pi_old_k )        then re-softmax

Worked on three classes trained at priors (0.5, 0.3, 0.2) and deployed where they are (0.2, 0.3, 0.5), on a row scoring (0.50, 0.20, 0.30). Multiply each probability by its prior ratio and renormalize:

prior ratio    0.2/0.5=0.4    0.3/0.3=1.0    0.5/0.2=2.5
reweighted     0.200          0.200          0.750     sum = 1.150
corrected      0.174          0.174          0.652

The argmax moved from class 1 to class 3. Softmax is invariant to adding the same constant to every logit and temperature scaling is monotone, so neither moves the argmax, but a prior correction adds a different constant per class, and that does. So prior correction is not calibration, and you owe it whether or not anything downstream reads the number.

Both forms of the correction as code:

from math import log

def prior_shift_correction(p_resampled, pi_true, pi_train):
    """Undo a change of class prior. Exact when p(x|y) is unchanged.
    Valid for random over/undersampling and class weights.
    NOT valid for SMOTE, which alters p(x|y=1) itself."""
    c = (pi_true / (1 - pi_true)) * ((1 - pi_train) / pi_train)
    odds = p_resampled / (1 - p_resampled)
    return (c * odds) / (1 + c * odds)

def logit_offset(pi_true, pi_train):
    """The same correction as a constant to add to every logit."""
    return log((pi_true / (1 - pi_true)) * ((1 - pi_train) / pi_train))

The SMOTE caveat is the practical payoff: the formula assumed f_1(x) was unchanged, which is what makes c a single constant. SMOTE violates it by construction (its synthetic points are new probability mass where no real positive sat), so its miscalibration varies from row to row, cannot be undone by an intercept, and must be repaired by fitting a calibrator on untouched held-out data (calibration).

Class weights are the exact objective; resampling is a noisy estimate of it

Weighting each positive row by w is, in expectation, identical to replicating it w times, but replication samples which rows get duplicated and undersampling discards rows, both introducing Monte Carlo noise (estimating a quantity by random sampling instead of computing it) that the weighted version does not have.

scikit-learn’s class_weight="balanced" sets w_c = n / (k · n_c), giving each class a weight inversely proportional to how common it is. On a 1% dataset (n = 10,000, n_1 = 100, n_0 = 9,900, k = 2): w_1 = 50.0, w_0 = 0.505, ratio 99, which is exactly (1-pi)/pi, the ratio that makes pi' = 0.5. So class_weight="balanced" and “resample to 50/50” are the same operation, break calibration the same way, and need the same correction. XGBoost’s scale_pos_weight is the same knob.

Two traps with the letter w:

• It is a different w from downsampling. In downsampling, w is a keep rate below 1 that shrinks the negatives, shifting every logit by +ln(w). Here, w is an up-weight ratio above 1 that inflates the positives, shifting every logit by -ln(w). The two are reciprocal prior moves (c = w versus c = 1/w), which is why one takes +ln and the other -ln. Carry the wrong one and you double the error instead of removing it. Whenever you see w, ask whether it multiplies rows or multiplies loss.
• w is the positive-to-negative weight ratio, not either weight alone. Only the quotient matters, since scaling both weights scales the whole loss and moves nothing. Feed w_1 = 50 into pi' = w·pi/(w·pi + 1 - pi) and you get 0.336; feed the ratio 99 and it lands on 0.500. At any prevalence c = 1/w: a model trained with scale_pos_weight = 12 needs -ln(12) = -2.485 added to every logit, no knowledge of pi required.

Which of the four to prefer:

Bagging (bootstrap aggregating) trains several copies of a model on different random subsets and averages them, cancelling much of any single copy’s noise.

Focal loss — rebalancing by difficulty instead of by class

Focal loss is a modified log loss that automatically pays less attention to examples the model already gets right, so the signal ends up dominated by hard cases instead of numerous easy ones. It uses p_t, “the probability the model assigned to the correct answer for this row” (high when right, low when wrong, whichever class the row is):

FL(p_t) = -alpha_t · (1 - p_t)^gamma · log(p_t)        p_t = p if y=1 else 1-p

-log(p_t) is ordinary log loss; alpha_t is an optional per-class weight; (1 - p_t)^gamma is the new part, near zero when p_t is near 1 and near 1 when p_t is near 0, so it shrinks the loss of easy examples and leaves hard ones almost alone. gamma = 0 recovers log loss; gamma = 2 is standard. At gamma = 2 an example at p_t = 0.9 contributes 100x less than the same at p_t = 0 and 81x less than a hard one at p_t = 0.1; at p_t = 0.99, 8,100x less.

Focal loss was invented for dense object detection: a detector lays a fixed grid of ~100,000 candidate boxes (anchor boxes) over an image, of which about 10 contain an object. Suppose easy background boxes sit at p_t = 0.99 and the positives at p_t = 0.10:

cross-entropy:
  100,000 easy negatives x -ln(0.99) = 1,005     background owns 43.6x more loss
       10 hard positives  x -ln(0.10) =    23

focal, gamma = 2:
  100,000 x 0.0001 x 0.01005 =  0.10             objects now own 186x more loss
       10 x 0.8100 x 2.30259 = 18.65

The gradient signal flips from 44:1 background to 186:1 foreground without touching the data (the gradient is a sum over rows, so whichever rows contribute most of the loss steer most of the learning). The flip is exactly the 8,100x modulation factor applied to the background’s share: 43.6 / 8,100 = 1/186.

Focal loss’s assumption is that your majority rows are overwhelmingly easy, not merely numerous. That is why it belongs to dense detection and rarely helps a tabular 1:99 problem: tabular negatives are genuinely hard to tell from positives, not sitting at p_t = 0.99. And focal loss is not a proper scoring rule: its minimizer is not p(y|x), so its output is a score to rank or threshold, not a probability to multiply by money. It happens to pull deep networks toward higher entropy (less peaked outputs), which usually reduces overconfidence, but that is a coincidence of direction, not a guarantee. If you need probabilities, calibrate afterwards.

When to do nothing

Often the correct action on an imbalanced dataset is no action. Do nothing when all four hold:

1. The metric is proper (log loss or Brier) or rank-based (AUC, PR-AUC).
2. The threshold came from a cost matrix instead of defaulting to 0.5.
3. Positives number in the thousands.
4. The downstream system consumes either a rank or a probability you have already corrected.

Resampling adds a distortion and a correction step; if you were not going to be wrong without it, you are adding two ways to be wrong.

The one thing “do nothing” does not excuse: a rank-only consumer still owes the prior correction: it just applies it to the cutoff instead of the scores. Adding ln(c) to every logit and subtracting ln(c) from the decision threshold select exactly the same rows. If a cost matrix gives an optimal threshold of 0.20 on true probabilities, then on a balanced-trained model at pi = 0.01:

ln c = -4.595,  logit(0.20) = -1.386
shifted = -1.386 - (-4.595) = 3.209      threshold on p' = sigmoid(3.209) = 0.961

Threshold the balanced-trained model at 0.961, not at 0.20: same rows, no probabilities touched. The two cutoffs differ by a factor of 99 in odds (which is 1/c). Take the 0.20 at face value and you flag almost everything. The sigmoid, 1/(1 + e^-z), is the function that undoes a logit.

Calibration: what it means, and when it matters

Everything above traded on calibration without pinning it down. A model is calibrated if, among all inputs it assigns probability q, exactly a fraction q are positive. Gather every row scored at 0.70; 70% of them should be positive.

for all q in [0,1]:   P(Y = 1 | p_hat(X) = q) = q

The hat on p_hat is standard notation for “an estimate of.”

Calibration is a weak property on its own. A model that outputs the base rate 0.03 for every input is perfectly calibrated and completely useless. The reliability - resolution + uncertainty decomposition of the Brier score makes that precise (Brier). The Brier score is the mean squared difference between the predicted probability and the 0-or-1 outcome, and like log loss it is a proper scoring rule. Its three terms:

• reliability is miscalibration, and you want it small.
• resolution is how far predictions spread away from the base rate, and you want it large.
• uncertainty is a property of the data alone, and you cannot change it.

A constant predictor has zero reliability and zero resolution: flawless on the calibration term, contributing nothing. So never report calibration without a discrimination number beside it. Discrimination is the other axis: the model’s ability to rank positives above negatives, measured by ROC-AUC, PR-AUC, or the resolution term. Calibration is about whether the number is right; discrimination is about whether the ordering is. A monotone recalibration fixes the first and cannot touch the second.

Reliability diagrams and ECE, with numbers

A reliability diagram (calibration curve) sorts predictions into bins by predicted value and plots, per bin, the mean predicted confidence against the fraction actually positive. A perfectly calibrated model traces the 45-degree line; a bin below the line is overconfident, one above is underconfident.

ECE, the expected calibration error, collapses that picture into one number: the vertical gaps averaged, weighted by how many rows each bin holds. Here are 10,000 predictions from an overconfident network, where every gap is positive, so every bin is overconfident:

ECE = sum_b (n_b/n)·|acc_b - conf_b|  = 0.1011
MCE = max_b |acc_b - conf_b|          = 0.125   (the 0.8-0.9 bin)
mean confidence - accuracy            = 0.898 - 0.797 = 0.101

The 0.9-1.0 bin holds 6,300 rows at a gap of 0.104, contributing 0.63 × 0.104 = 0.0655, nearly two thirds of the total. MCE, the maximum calibration error, is the single worst bin; being the sparsest bin as often as the worst, it reports noise about as often as a problem.

When miscalibration runs in one direction, ECE equals |mean confidence - accuracy|: one line of code, no binning decision, the fastest overconfidence check there is.

Running all three Brier terms over the same table shows how little of the story ECE carries. Writing pbar for the overall positive rate:

pbar        = 0.797
uncertainty = pbar(1 - pbar)             = 0.161666
reliability = sum_b (n_b/n)(p_b - o_b)^2 = 0.010585
resolution  = sum_b (n_b/n)(o_b - pbar)^2 = 0.012185
Brier = reliability - resolution + uncertainty = 0.160065

reliability is the mean squared gap; ECE is the mean absolute gap, the same thing in different units (sqrt(0.010585) = 0.103 against ECE = 0.1011), coinciding only because every bin errs the same way. Now the comparison ECE cannot make. A constant predictor emitting 0.797 scores Brier 0.161666 (the uncertainty term alone). This model scores 0.160065. It beats a constant by 1%. Recalibrate it perfectly and reliability drops to zero, landing Brier at 0.149480, a 6.6% gain. Recalibrating buys 0.010585 of Brier; owning the model at all instead of a constant buys 0.001601, a factor of 6.6 smaller. The confidence numbers are badly wrong and the ranking is barely there, and no ECE value could tell you the second half of that.

ECE’s binning sensitivity, and why it is a lower bound

ECE is a property of the binning choice, not of the model, but only sometimes. Equal-width binning cuts [0,1] into equal slices (bins hold different counts); equal-mass binning makes every bin hold the same number of rows (slices have different widths). Run the same 10,000 predictions through 5, 10, 15 equal-width bins and 10 equal-mass bins and all four return 0.1011.

The reason: merge two bins A and B and the merged bin reports |w_A·g_A + w_B·g_B|, where g is a signed gap and w a row share; the separate bins reported w_A·|g_A| + w_B·|g_B|. These are equal whenever the two gaps carry the same sign. Every bin of this model is overconfident, so no regrouping moves the total off |0.898 - 0.797| = 0.1011. No rebinning of an everywhere-overconfident model can change its ECE.

Bin count starts mattering the moment a single bin holds errors of opposite sign, which happens once one wide bin over [0.0, 0.5) holds two groups:

100 predictions at 0.10, of which 40 are positive   (true rate 0.40)  underconfident
100 predictions at 0.40, of which 10 are positive   (true rate 0.10)  overconfident

bin mean confidence = 0.25    bin observed rate = 0.25    gap = 0.000

ECE contribution: zero. Actual miscalibration: 0.30 on every one of those 200 rows. Within-bin errors of opposite sign cancel, and coarser bins give them more room to cancel. Add one bin edge at 0.25 and each half reports gap 0.30, weight 0.5, so ECE jumps to 0.300, moved from 0.000 to 0.300 without changing a single prediction.

Two consequences:

• ECE is a lower bound on miscalibration, monotone in bin coarseness. “Our ECE is 0.02” is not a claim until the bin count and bin-mass policy are stated with it.
• ECE is not a proper scoring rule and can be gamed: a model can lower its ECE while getting worse. Report ECE for interpretability and a proper score (log loss, Brier) for the decision.

The implementation is short so you can see where the binning decision enters, since both n_bins and equal_mass change the answer on some models and not others, which is exactly why they belong in the number you report:

def ece(probs, labels, n_bins=10, equal_mass=False):
    """Expected calibration error. Report n_bins and the binning mode with it."""
    pairs = sorted(zip(probs, labels))
    n = len(pairs)
    if equal_mass:
        edges = [i * n // n_bins for i in range(n_bins + 1)]
        groups = [pairs[edges[i]:edges[i + 1]] for i in range(n_bins)]
    else:
        groups = [[] for _ in range(n_bins)]
        for p, y in pairs:
            groups[min(int(p * n_bins), n_bins - 1)].append((p, y))
    total = 0.0
    for g in groups:
        if not g:
            continue
        conf = sum(p for p, _ in g) / len(g)
        acc = sum(y for _, y in g) / len(g)
        total += (len(g) / n) * abs(acc - conf)
    return total

Platt scaling vs isotonic regression

Two calibrators are standard, each resting on its own assumption. Both fit a monotone map from raw score to probability on held-out data (a map that never reorders two rows), so neither can change AUC beyond tie-breaking. They differ in how much shape they can express.

PAVA is the pool-adjacent-violators algorithm, which fits isotonic regression by repeatedly merging any neighbouring pair that runs the wrong way. NLL, negative log-likelihood, is log loss under another name. One-vs-rest fits one binary model per class and combines the answers. s in Platt’s formula is the raw score (usually the logit), with a and b the two fitted numbers.

The most decision-relevant entry is “Platt can be worse than doing nothing.” Take a model exactly right at the bottom of its range and badly overconfident only at the top, an asymmetric distortion, which is exactly the case Platt’s sigmoid excludes. 1,000 held-out rows at three score levels, only the top one wrong:

n = 400   raw 0.10   true rate 0.10      gap 0.000
n = 400   raw 0.50   true rate 0.50      gap 0.000
n = 200   raw 0.90   true rate 0.55      gap 0.350       raw ECE = 0.2 x 0.350 = 0.0700

Fitting Platt means choosing a, b to minimize log loss on these rows. A two-parameter sigmoid cannot bend only at the top, so it tilts everywhere:

fitted a = 0.5382, b = -0.4822
  0.10 -> 0.159   gap 0.059   (was 0.000)
  0.50 -> 0.382   gap 0.118   (was 0.000)
  0.90 -> 0.668   gap 0.118   (was 0.350)
Platt ECE    = 0.0946   <- worse than the 0.0700 it started from
isotonic ECE = 0.0000   <- three levels already monotone; PAVA reproduces them exactly

Platt cut log loss from 0.6261 to 0.5683 (it is fitted to do precisely that) while raising calibration error from 0.0700 to 0.0946, by smearing one bin’s error across the 800 rows that were already exact. Two parameters buy low variance (the fit barely moves if you resample the calibration set) and pay in bias (the shape is systematically wrong no matter how much data you give it). When the true distortion is not a sigmoid, that bias lands on the rows that were already correct. Isotonic assumes only monotonicity, so it absorbs the shape instead.

So the choice is between assumptions: Platt assumes the distortion is a sigmoid and needs a few hundred rows; isotonic assumes only that the distortion never reorders anything and needs a few thousand.

The two multiclass forms are not symmetric. Temperature scaling fits one scalar across all K logits, so the output stays a distribution by construction. One-vs-rest isotonic fits K independent step functions that need not sum to 1, so you renormalize afterwards, itself an uncalibrated step that can undo part of each fit. That asymmetry, not accuracy, is why temperature is the deep-learning default.

PAVA worked by hand

Eight held-out rows, sorted by model score, with true labels [0, 1, 0, 0, 1, 1, 0, 1]. Isotonic wants fitted values that never decrease left to right. PAVA repeatedly finds a decreasing neighbouring pair (a “violator”) and replaces both with their average:

start                       0 | 1 | 0 | 0 | 1 | 1 | 0 | 1
merge 1,0 at positions 2,3  0 | 0.500 | 0 | 1 | 1 | 0 | 1
merge with the 0 at pos 4   0 | 0.333 | 1 | 1 | 0 | 1
merge 1,1,0 at 5,6,7        0 | 0.333 x3 | 0.667 x3 | 1
result: [0, 0.333, 0.333, 0.333, 0.667, 0.667, 0.667, 1.0]   monotone, done

Positions 2, 3 and 4 held three distinct raw scores and all collapsed to 0.333. That is the ties problem: inside a pooled block, ranking information is destroyed, which is why isotonic can shave a fraction off AUC even though it is monotone.

Temperature scaling

You fit a single positive number T by minimizing log loss on a validation set, then divide every logit by it before the softmax. T > 1 flattens the distribution and reduces confidence; T < 1 sharpens it. Because z/T is strictly increasing in z, argmax, accuracy, and AUC are all unchanged: only the numbers change. With one parameter fitted on thousands of rows it essentially cannot overfit. Its assumption (that the whole logit vector is miscalibrated by one common factor) is strong, and it happens to describe the deep-net mechanism below almost exactly.

The rule that catches everyone

The calibrator must be fitted on data the model did not train on. A calibrator fitted on training predictions learns the model’s training-set confidence, which is far higher than its confidence on new inputs, so you “calibrate” the model into being worse. Use a dedicated held-out calibration split. If data is scarce, use cross-fitted calibration: split into folds, fit the model on all but one, collect that fold’s out-of-sample predictions, repeat, and fit one calibrator on the pooled out-of-sample predictions.

Why modern deep nets are overconfident — the mechanism

Why does a large network reliably claim more confidence than it has earned? A chain of steps, forced one by the next:

flowchart TD
    A["Training error reaches zero"] --> B["Every training point is<br/>on the correct side"]
    B --> C["NLL is still positive,<br/>and scaling all logits by k > 1<br/>strictly lowers it"]
    C --> D["No finite minimizer:<br/>the optimizer keeps growing<br/>the logit magnitude"]
    D --> E["Softmax saturates<br/>-> confidence near 1.0<br/>on training inputs"]
    E --> F["That confidence transfers<br/>to test inputs where the model<br/>is right only 80% of the time"]
    F --> G["mean confidence 0.90<br/>accuracy 0.80<br/>ECE 0.10"]

    style C fill:#bc6c25,color:#fff
    style D fill:#9d0208,color:#fff
    style G fill:#1d3557,color:#fff

Once training error reaches zero, every point sits on the correct side of the boundary (classification is finished), yet NLL is still positive, because -log p vanishes only in the limit. That leaves one direction the loss can still fall: scale all logits by k > 1. It moves no point across the boundary and strictly lowers NLL. So there is no finite minimizer, and the optimizer keeps growing the logit magnitude until the softmax saturates. That confidence is a property of the weights, not the training set, so it transfers intact to test inputs where the model is right only 80% of the time, arriving as mean confidence 0.90 against accuracy 0.80.

The load-bearing step is one line of arithmetic, the same decision, scaled up by 3:

logits (2, 0) -> sigmoid(2) = 0.881 -> loss 0.127
logits (6, 0) -> sigmoid(6) = 0.998 -> loss 0.003     same decision, same ranking, 51x lower loss

Three modern practices make this worse: capacity large enough to reach zero training error is routine; weight decay (a penalty on weight size, which would otherwise stop the logits growing) is dialled down because it costs accuracy; and residual connections and normalization make reaching zero training error easy. So overconfidence is the predicted outcome.

The fix, fitting a temperature of 1.9 on this network:

                accuracy   mean conf   ECE     after T = 1.9
test               0.797       0.898   0.1011   ECE 0.0121, accuracy 0.797

Accuracy did not move (it cannot, because temperature is monotone), while the calibration error fell by a factor of eight for the cost of fitting one number.

Why boosted trees are under-confident at the extremes

Boosted trees fail the opposite way, by a different mechanism, so the fix differs too. A boosted model is additive: F = F_0 + eta · sum_m h_m, starting from a base score F_0 and adding each tree’s output scaled by a learning rate eta deliberately kept small (0.05 is typical). Every leaf carries w = -G/(H + lambda), where G is the summed loss gradient over the leaf’s rows, H the summed second derivative, and lambda an L2 regularization term that keeps leaf values small (XGBoost’s split criterion). So each leaf value is shrunk twice: by lambda inside and eta outside.

To emit p = 0.99 the model needs F = ln(99) = 4.595. At eta = 0.05, and because the step eta·w collapses as p -> 1 (both G and H go to zero, so w -> n(1-p)/lambda), that takes a run of rounds that grows fast per extra nine of confidence:

rounds to reach p = 0.50      19
rounds to reach p = 0.90      61
rounds to reach p = 0.99     198       <- the last 0.09 of probability costs 137 rounds
rounds to reach p = 0.999  1,146
rounds to reach p = 0.9999 10,193
same run to p = 0.99 at lambda = 0 -> 75 rounds;  at lambda = 5 -> 631

lambda is negligible against H in the middle of the range and becomes the whole denominator at the top, which is why it alone stretches the run from 75 to 631 rounds without changing the fit’s direction. And early stopping (which halts when validation loss stops improving) watches the many rows in the middle, so it cuts the run off in the flat part long before the confident rows saturate.

min_child_weight closes the other door: a floor on H = sum_i p_i(1-p_i) within a leaf, the exact quantity that collapses as predictions saturate. Per-row h is 0.25 at p = 0.50 but only 0.0099 at p = 0.99, so a leaf needs 4 rows to clear the XGBoost default of H >= 1 in the middle and 102 near the top. The regularizer that keeps boosting from overfitting is the same one that keeps it from ever saying 0.99. The result is the classic sigmoid reliability curve: too high at the bottom, honest in the middle, too low at the top, with the model never emitting above ~0.961. That shape is exactly what Platt scaling inverts, which is why Platt is the traditional recommendation for boosted trees and for SVMs (support vector machines, which output an uncalibrated distance from a boundary instead of a probability).

The production failure is silent: a rule “auto-block when p > 0.99” never fires, because the maximum output is 0.961. Nothing errors; the rule is simply dead until someone plots the score histogram.

Random forests are under-confident too, for an unrelated reason

The reason is averaging. A random forest grows B decision trees, each on a bootstrap resample of the rows (a sample drawn with replacement) and each allowed to consider only m randomly chosen features per split. The output is the fraction of trees that voted positive, so emitting exactly 1.0 requires unanimity, and how hard unanimity is depends on how correlated the trees are.

Trees in a forest are not independent, because they share the same data. The variance of the vote can never fall below rho·sigma_t^2 no matter how many trees you add (the variance floor), where rho is the average correlation between two trees’ votes. Feature sampling controls rho: about 0.35 when every feature is available at every split, 0.12 at m = sqrt(p), 0.03 at m = 1 (decorrelation).

Model the votes as exchangeable (any two share the same correlation rho) with a beta-binomial, and the independence figure people quote is not just imprecise, it points at the wrong rows:

P(all 100 trees vote positive), per-tree accuracy 0.95
   rho = 0 (independent)   0.95^100 = 0.0059
   rho = 0.03                      = 0.097
   rho = 0.12                      = 0.362
   rho = 0.35                      = 0.669
per-tree accuracy 0.75:   rho = 0 -> 3e-13,   rho = 0.12 -> 0.005

On an easy row a real forest reaches 1.0 between a third and two thirds of the time: the extremes are not bounded away at all. Where averaging genuinely traps the output is on borderline rows: at per-tree accuracy 0.75 the forest lands near 0.75 and essentially never at 1.0, because bootstrap resampling keeps a persistent dissenting minority no matter how many trees you add. The variance of the vote fraction is p(1-p)·(rho + (1-rho)/B); drive B to infinity and it settles at p(1-p)·rho, a standard deviation near 0.15 at rho = 0.12. Correlation multiplies the spread by 3.6 over the independent 0.0433 and still leaves most of the mass in the interior. Borderline rows are exactly where a decision threshold sits, which is why this costs money. The fix is the same as boosting’s: fit a monotone calibrator, or read the output as a rank.

When calibration matters, and when it is irrelevant

The rule: calibration matters exactly when something reads the number’s face value instead of its position in an ordering. NDCG in the first row (normalized discounted cumulative gain, a ranked-list quality measure) depends only on order, like AUC.

If nothing downstream reads the number, calibration is a metric you can afford to lose. If anything multiplies it, compares it to a constant, or shows it to a person, it is the whole game.

Drift: three different failures with three different signals

Drift is the general name for live data drifting away from the training data, the reason a model’s accuracy decays over time even though nobody touched it. There are three kinds, needing three different fixes, and only two can be detected before labels arrive.

The taxonomy is not arbitrary. Factor the joint distribution p(x, y) (the probability of an input together with a label) the two ways it can be factored. Each factoring isolates a pair of things that can move independently, and every drift type is one piece moving while its partner stays put:

p(x, y) = p(y | x) · p(x)          <- covariate shift lives here
        = p(x | y) · p(y)          <- label shift lives here

• Covariate shift is p(x) moving while p(y|x) holds: the inputs changed but the input-to-answer relationship did not, so a customer with a given profile is still just as risky. (A covariate is an input feature.)
• Label shift is p(y) moving while p(x|y) holds. The class mix changed but what each class looks like did not: twice as much fraud, and fraud still looks like fraud. Also called prior shift, since p(y) is the prior from resampling.
• Concept drift is p(y|x) itself moving: the same input now deserves a different answer. No reweighting repairs this.

flowchart TD
    D["Model degraded in production"] --> Q1{"Did p of x move?"}
    Q1 -->|yes| CS["COVARIATE SHIFT<br/>inputs moved, p of y given x fixed"]
    Q1 -->|no| Q2{"Did the predicted<br/>positive rate move?"}
    Q2 -->|yes| LS["LABEL SHIFT<br/>the prior moved, p of x given y fixed"]
    Q2 -->|no| CD["CONCEPT DRIFT<br/>the relationship itself changed"]
    CS --> F1["Detect: feature PSI,<br/>domain classifier - NO LABELS<br/>Fix: importance weighting,<br/>retrain, drop unstable features"]
    LS --> F2["Detect: predicted-rate monitor,<br/>BBSE - NO LABELS<br/>Fix: constant logit shift"]
    CD --> F3["Detect: rolling loss on<br/>delayed or proxy labels - LABELS REQUIRED<br/>Fix: retrain. Nothing else works."]

    style CS fill:#2d6a4f,color:#fff
    style LS fill:#40916c,color:#fff
    style CD fill:#9d0208,color:#fff

The first question (did p(x) move?) is answerable immediately and without labels, and splits covariate shift from the other two. The second (did the predicted-positive rate move?) splits label shift from concept drift. Concept drift is the residual: every distribution you can see is stable and the model is wrong anyway. The three side by side:

Why covariate shift hurts at all

In theory it should not. If the model were correctly specified (the true relationship is in the family it can express) and well-estimated everywhere, covariate shift would cost nothing, since p(y|x) is unchanged. It hurts for two concrete reasons:

1. The model is misspecified, so the fit was a compromise across regions weighted by the old p(x), which has now changed.
2. The new inputs land where there was little training data, so the model is extrapolating.

So “PSI is high” is not by itself a reason to retrain. It is a reason to check whether the moved mass sits in a region the model was ever good at.

The three covariate-shift fixes

Drop unstable features needs a criterion or it is a slogan. The instability that earns a drop is high PSI and meaningful permutation importance (how much accuracy falls when you shuffle one feature’s values) and no reason to expect the movement to stop. A feature that drifts and the model leans on is a scheduled outage; one that drifts and the model ignores is noise in your alerting. The trade is permanent: dropping a moved feature costs accuracy on the population that did not move, to buy stability against the part that did. Take it when the drift is structural (an upstream vendor you do not control, a field redefined every release), not for a one-off a retrain would handle.

Importance weighting retrains or re-evaluates on the old data with each row weighted by how much more common it is now, so the old data stands in for the new distribution. Estimating p_new(x) and p_old(x) directly and dividing is hopeless in more than a few dimensions. Instead train a domain classifier d(x) = P(x came from the new period) (an ordinary binary classifier whose label is which period the row came from) and convert its odds to the density ratio by Bayes’ rule:

p_new(x)/p_old(x) = [ d(x)/(1 - d(x)) ] · (n_old / n_new)

With equal-sized periods the correction factor is 1 and the weight is just the classifier’s odds: d(x) = 0.75 means the region is 3x as common now; d(x) = 0.90 means 9x. The n_old/n_new factor matters the moment the periods differ in size: score four weeks of history against one week of production and the factor is 4, so d(x) = 0.50 means “as many new rows as old land here,” which with 4x fewer new rows overall is a 4x enrichment. For a reweighted retrain a constant factor is harmless (it scales the whole loss), but the moment you read a weight as “how much more common,” dropping the ratio is the difference between 1x and 4x.

Importance weighting assumes the new distribution’s support is contained in the old one: every region with new rows had at least some old rows. Where that fails the density ratio is infinite and a handful of old rows carry the entire retrain.

The same classifier is a drift detector for free. Its held-out AUC is the drift magnitude (0.5 means the periods are indistinguishable, 0.85 means severe), and its feature importances name which features moved. One model gives the alarm, the diagnosis, and the correction weights. The two AUC figures this chapter quotes are not in conflict: 0.70 is where you look; 0.85 is where you act. At 0.85 the classifier can nearly separate the periods, meaning most new rows sit where the training data barely reached, so importance weighting is running on ratios estimated from almost no old data.

Label shift, and why it is the §2 formula again

Label shift is a change of prior with p(x|y) fixed (the identical assumption behind resampling), so the identical correction applies, with old and new prevalences in place of training and true:

logit(p_new) = logit(p_model) + ln( [pi_new/(1-pi_new)] · [(1-pi_old)/pi_old] )

The only new problem is estimating pi_new with no labels from the new period. BBSE, black-box shift estimation, does it from predictions alone, using two numbers you measured on the old validation set at your operating threshold: TPR (the fraction of actual positives flagged) and FPR (the fraction of actual negatives flagged by mistake). Both are properties of p(x|y), which label shift leaves alone, so they carry over unchanged. The predicted-positive rate is then a linear function of the unknown prevalence, since every flagged row is a true positive (pi fraction, caught at TPR) or a false positive (1-pi fraction, flagged at FPR):

q = P(predict positive) = TPR·pi + FPR·(1 - pi)

You know q, TPR, FPR; solve for pi. Worked with TPR = 0.80, FPR = 0.05, pi_old = 0.10, and 2,300 of 10,000 new rows flagged (q = 0.23):

0.23 = 0.80·pi + 0.05·(1 - pi) = 0.05 + 0.75·pi   ->   pi_new = 0.18/0.75 = 0.240
logit shift = ln( (0.240/0.760)·(0.900/0.100) ) = ln(2.842) = 1.0445
a row scoring 0.30: odds 0.4286 -> 0.4286 × 2.842 = 1.218 -> p = 0.549

You corrected the model today, from predictions only, while the true labels are still 45 days out. The alternative takes two months: wait for chargebacks (the disputed-transaction reversals that are a card issuer’s ground-truth fraud label, six weeks to settle), notice accuracy fell, retrain.

The assumption you are betting on is that p(x|y) held. If the attackers’ method changed and not merely their volume, you have concept drift instead: the TPR and FPR you carried over are stale and the correction is wrong in an unknown direction. Sanity-check before applying: confirm feature PSI stayed low while the predicted-positive rate moved. Same-looking inputs plus more positive predictions is genuine label shift; moved inputs means BBSE’s assumption has failed.

PSI and KL, computed

PSI, the population stability index, answers one question about one feature: how far has its distribution moved since training? Cut the feature’s range into bins whose edges are frozen at training time (usually the training deciles, the ten cut points splitting the training data into equal tenths), then compare each bin’s share now against its share during training:

PSI = sum over bins of  (a_i - e_i) · ln(a_i / e_i)

e_i is the expected share (bin i’s share of training rows) and a_i the actual share today. Because the bins were cut at training deciles, every e_i is 0.10 by construction. Freezing the edges is the entire method: re-binning on production data would compare a distribution against itself and always return roughly zero.

A feature that has drifted upward (low bins emptied, high bins filled):

bin    e_i    a_i    (a-e)·ln(a/e)
 1     0.10   0.03      0.084278       <- one bin, 38% of the total
 2     0.10   0.05      0.034657
 3     0.10   0.06      0.020433
 ...
10     0.10   0.16      0.028200
                    PSI = 0.219344

Conventional thresholds: < 0.10 stable, 0.10-0.25 investigate, > 0.25 act. So this feature lands in “investigate.” But those thresholds are folklore, and three facts explain why.

Fact 1: PSI is symmetric KL divergence. KL divergence measures how much information you lose by using one distribution in place of another: zero when they match, positive otherwise, and asymmetric. Splitting the PSI sum (using -ln(a/e) = ln(e/a)):

sum (a - e)·ln(a/e) = KL(a || e) + KL(e || a)          <- Jeffreys divergence
here:  0.219344 = 0.1011 + 0.1182

Every term is non-negative, so PSI is zero exactly when the two distributions match. It is a symmetrized information distance with a folklore threshold table bolted on.

Fact 2: it is dominated by the bins that emptied, not the bins that filled. Bin 1 alone contributes 38% of the total from 3% of the mass, because ln(a/e) runs to negative infinity as a -> 0 but grows only logarithmically as a grows. A completely empty bin would make PSI infinite, so every implementation floors a_i at some epsilon (0.0001 is common). That arbitrary epsilon silently sets your alert threshold: a feature with one empty bin reports whatever the epsilon dictates.

Fact 3: the thresholds ignore sample size, which cannot be right. Even with no drift, a finite sample will not reproduce the training shares exactly, so PSI is positive by luck. Under pure multinomial sampling noise (throwing n rows into K bins), 2n·KL follows a chi-square with K-1 degrees of freedom, whose mean is K-1, so one KL direction has mean (K-1)/(2n). PSI is both directions, equal to leading order, so the floor is twice that:

E[PSI]  ~  2 · (K-1)/(2n)  =  (K-1)/n

Simulation confirms the doubled version: 200,000 no-drift draws at K = 10, n = 10,000 give mean PSI 0.0009, matching 9/10,000. Reading the floor off at three batch sizes:

K = 10, n = 10,000  ->  E[PSI] ≈ 0.0009    (negligible)
K = 10, n =    500  ->  E[PSI] ≈ 0.018
K = 10, n =    100  ->  E[PSI] ≈ 0.090     <- already at "investigate", from noise alone

On a small daily slice the 0.1 threshold fires on pure noise; on 10 million rows a PSI of 0.05 is significant and operationally meaningless. The fix is to calibrate the threshold to your own batch size: bootstrap the PSI. Resample stable historical windows at the batch size you actually score, recompute PSI many times, and set the alert at the p99 of that distribution.

Two limits PSI cannot check about itself: it is univariate, so income and debt can each hold a calm PSI of 0.02 while their joint relationship inverts and a model that splits on both is destroyed with every monitor green; and it is blind to importance, so a drifting feature the model barely uses pages you for nothing. Weight PSI by permutation importance to fix the second, and use the domain classifier (which is inherently multivariate) to fix the first.

from math import log

def psi(expected_counts, actual_counts, eps=1e-4):
    """PSI = KL(a||e) + KL(e||a) over fixed bins.
    Bin edges must be frozen at training time; re-binning on production
    data compares a distribution against itself and returns ~0."""
    e_tot, a_tot = sum(expected_counts), sum(actual_counts)
    total = 0.0
    for e_c, a_c in zip(expected_counts, actual_counts):
        e = max(e_c / e_tot, eps)          # eps here sets your alert threshold
        a = max(a_c / a_tot, eps)
        total += (a - e) * log(a / e)
    return total

def psi_noise_floor(n_rows, n_bins):
    """E[PSI] under no drift: (K-1)/n. PSI is both KL directions, equal to
    leading order, so the floor is twice the single-direction (K-1)/(2n)."""
    return (n_bins - 1) / n_rows

Detecting drift without labels — the real production problem

In a running system, the thing you most want to measure (whether the model is still right) is the thing you cannot have. Labels arrive late (chargebacks take 45-60 days), partially (only approved loans produce a repayment outcome), or never (nobody writes in to say a recommendation was bad). Every drift signal that arrives in time needs no labels, and none of them can see concept drift.

Read the menu as a ladder: each row is slower and more informative than the one above, and the Misses column is why you cannot rely on any single one. Latency is how long after a problem starts the signal can tell you; p95 is the 95th percentile of the score, watched alongside the mean because a distribution can shift in its tail while its average holds.

The embedding row is the only signal here that works on unstructured data, where PSI cannot (a sentence or image has no scalar to bin). An embedding is a fixed-length vector the model’s encoder produces as its internal summary of an input, in which similar inputs land near each other. Freeze a reference sample of training embeddings at release (10,000-100,000 vectors) and monitor how far each production batch sits from it, by mean cosine distance (the angle between vectors, ignoring length) to the k nearest reference vectors, Mahalanobis distance (Euclidean distance rescaled by how much the reference varies in each direction), or maximum mean discrepancy (comparing the two clouds as wholes). This catches the input that is unremarkable in every individual coordinate but jointly sits somewhere the model has never been: a new language, a new template, a camera with a different colour response. It still misses the in-domain input whose correct answer changed.

You cannot buy your way out of concept drift with a better detector. The information is not there, because p(y|x) is what changed and y is what you do not have. So the investment that pays is engineering a fast proxy label: a click at 60 seconds that correlates 0.7 with a conversion at 30 days converts a two-month detection latency into a one-day one, and no amount of input monitoring does that.

Feedback loops: the model changes the distribution it is measured on

One failure evades every monitor above, because p(x) never moves. What moves is which labels you are allowed to observe: the model’s own decisions determine which outcomes get recorded. One fraud pattern P through a single cycle:

week 0   model blocks 95% of pattern P.  P is 2.0% of labeled fraud.
week 4   labeled data now holds only the 5% that slipped through;
         P's apparent share of labeled fraud has fallen to 0.4%.
week 6   scheduled retrain. P's learned risk drops from 0.71 to 0.28.
week 7   deploy. Threshold is 0.30, so P now passes cleanly.
week 9   losses on P up 6x.  Feature PSI across every input: 0.03. No alarm.
week 14  chargebacks settle; aggregate accuracy finally moves 0.4 points.

The model suppressed the evidence for its own decision, and the retrain believed the evidence. Credit scoring has the same structure (only approved applicants produce a repayment outcome, and guessing what the rejected ones would have done is reject inference), as does every recommender (an item never shown accumulates no clicks and looks permanently irrelevant, why offline ranking metrics disagree with online CTR).

The fix is structural, because no cleverness recovers information that was never recorded:

• Randomized hold-back. Deliberately let 1-2% of would-be blocks through, logged as an experiment. It costs a budgeted amount of fraud and buys an unbiased stream of labels covering the rows the model would otherwise hide from itself. This is the only fix that actually works.
• Log the propensity of every action (the probability the system would have taken it, which you know because you chose the policy), so you can reweight the historical log later by inverse propensity, giving rarely-taken actions proportionally more weight.
• Monitor the action distribution, not just the input distribution: block rate, approval rate, score distribution per segment (any slice you can name in advance: a country, a device type, a merchant category). In the trace, P’s block rate moved in week 7, seven weeks before the accuracy number did.

Monitoring design, and why aggregate accuracy is too slow

Overall accuracy on real labels is the wrong thing to alert on, for three independent reasons:

1. Label latency. If chargebacks settle at 45 days, accuracy computed today reports on the model as it was a month and a half ago, too late to act on.
2. Dilution. An aggregate is a weighted average, so a catastrophe in a small slice barely registers. A segment holding 5% of traffic collapsing from 0.92 to 0.50 accuracy, with everything else steady at 0.91, moves the headline only from 0.9105 to 0.8895, 2.1 points for a total failure in a real segment.
3. Statistical power: a test’s ability to notice a real effect. Detecting a 1-point drop from 0.91 at two standard errors, using se_diff = sqrt(2·p(1-p)/n) and setting 2·se_diff = 0.01, needs n ≈ 6,552 labeled rows per period. At 500 labeled rows a day that is 13 days per period, 26 days to compare two.

A one-point regression takes about a month to become statistically visible and is diluted 20:1 if it lives in one segment. Aggregate accuracy is an audit metric, not an alert metric.

Alert on a ladder instead, cheapest and fastest first:

flowchart TD
    P["PIPELINE<br/>schema · nulls · ranges<br/>latency: minutes<br/>certainty: absolute"] --> I["INPUT<br/>feature PSI · domain classifier<br/>latency: hours<br/>certainty: statistical"]
    I --> O["OUTPUT<br/>predicted-positive rate · mean score<br/>mean confidence<br/>latency: hours"]
    O --> AC["ACTION<br/>block and approval rate<br/>PER SEGMENT<br/>latency: hours"]
    AC --> PX["PROXY OUTCOME<br/>fast label standing in<br/>for the slow one<br/>latency: days"]
    PX --> T["TRUE OUTCOME<br/>log loss · AUC on real labels<br/>latency: weeks to months<br/>AUDIT, never a page"]

    style P fill:#2d6a4f,color:#fff
    style PX fill:#40916c,color:#fff
    style T fill:#9d0208,color:#fff

The diagram gives the order; the table adds the cadence and the rule:

Sigma is a standard deviation, so “3 sigma vs the trailing 4 weeks” means further from the recent norm than three standard deviations, about once in 370 observations under ordinary noise. Hour-of-week comparison pits Tuesday 3 p.m. against previous Tuesdays at 3 p.m., so weekly seasonality does not read as drift.

Alert on the pipeline before the model. The most common cause of a “drift” alert is an upstream change, catchable in minutes with certainty, not in weeks with statistics:

09:14  upstream job changes `days_since_signup` from days to seconds
09:15  null rate 0.0% (no alarm); range check FAILS: max 315,360,000 vs training max 3,650
09:15  range alert -> rollback.  Total exposure: 1 minute.
without the range check:  next day feature PSI = 4.1 (15 hours later);  +45 days accuracy moves

The unit change made the field 86,400x larger, which the range check catches instantly. PSI needs a day; accuracy needs 45.

Prefer scheduled retraining on a cadence shorter than your drift timescale over drift-triggered retraining: triggered retraining chases noise and creates a feedback loop of its own, since each retrain is fitted on data the previous model shaped. Whatever the cadence, compare the candidate against the incumbent on two holdouts: a frozen historical one that catches regressions, and a fresh recent one that catches staleness. A candidate that wins on the fresh set and loses on the frozen one has learned the drift, which may be exactly what you wanted, or overfitting to a two-week anomaly, and the two look identical until you ask which it is.

Cheat sheet

A symptom on the left, the mechanism in the middle, the fix on the right:

Conclusion

1. Imbalance, resampling, class weights, and label shift are all the same prior moving, and all are repaired by logit(p) = logit(p') + ln(c), an exact intercept shift that leaves ranking untouched. It is not calibration, and you owe it whenever anything reads the number or a probability-unit threshold.
2. Calibration is a separate axis from ranking. Deep nets grow overconfident because at zero training error growing the logits is the only way NLL can fall; boosted trees and forests stay under-confident at the extremes through shrinkage and averaging. All are repaired by a monotone map that cannot change AUC, and calibration is worth nothing unless something downstream reads the number.
3. Drift splits into covariate shift, label shift, and concept drift; the practical distinction is that the first two are detectable without labels and the third (the dangerous one) is not. So the thing to build is a fast proxy label, not a better detector, and the alert ladder runs cheap-and-certain (pipeline checks) before slow-and-statistical (accuracy).

One line to remember: a classifier’s ranking and its probabilities are separate objects, damaged by separate things and repaired by separate tools, so the first question every time is whether anything downstream reads the number or only the order.

Further reading

• Guo, Pleiss, Sun, Weinberger, “On Calibration of Modern Neural Networks” (2017): the zero-training-error overconfidence mechanism and temperature scaling.
• Lin, Goyal, Girshick, He, Dollár, “Focal Loss for Dense Object Detection” (2017): focal loss and the dense-detection regime.
• Chawla, Bowyer, Hall, Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique” (2002): the original method and its interpolation assumption.
• Niculescu-Mizil, Caruana, “Predicting Good Probabilities with Supervised Learning” (2005): Platt scaling versus isotonic regression, empirically.
• Lipton, Wang, Smola, “Detecting and Correcting for Label Shift with Black Box Predictors” (2018): the BBSE estimator.

Next: 08 — Reinforcement Learning: decisions that change the distribution they are evaluated on, taken deliberately this time.