Imbalanced Data, Calibration, and Drift

A classifier’s ranking (the order of its scores) and its probabilities (the numbers at face value) are separate objects, broken by separate things and repaired by separate tools, so the first question is always whether anything downstream reads the number or only the order.

The one idea underneath

• Imbalance, resampling/class weights, and label shift are all one thing: a prior p(y=1) that moved while p(x|y) (what each class looks like) held.
• All are repaired by the same prior-shift correction in log-odds space: logit(p) = logit(p') + ln(c), where logit(p) = ln(p/(1-p)).
• The shift is a monotone map: AUC is unchanged (rank-only), every probability is fixed. It is not calibration.
• c = [pi/(1-pi)] · [(1-pi')/pi'] (true prior over training prior).

Imbalance triage

Usually a threshold/metric problem downstream, not a data problem. Ask in order:

• Absolute positives < ~1,000 → real data problem (p(x|y=1) has no support). Get labels, simpler model, anomaly detection. (~1,000 from ~10–20 events/coefficient and metric noise 1/sqrt(n_pos); not a constant.)
• Thousands+ → what actually broke:
  • Metric is accuracy (prevalence-weighted) → use PR-AUC, MCC, or expected cost.
  • Decisions use argmax at 0.5 → threshold from cost: t* = C_fp/(C_fp+C_fn).
  • Optimizer drowned by easy negatives → class weights or focal loss.
  • Nothing → log loss (a proper scoring rule) fits a low-prior model fine; do nothing.

The four techniques

• Prefer class weights almost always. class_weight="balanced" sets ratio (1-pi)/pi = “resample to 50/50” and needs the same correction. XGBoost scale_pos_weight is the same knob; there c = 1/w.
• SMOTE fails on non-convex/multi-modal minority, categorical (one-hot) features, and high dimensions; also leaks if applied before the CV split (resample strictly inside the fold).
• Negative downsampling (keep each negative with prob w): c collapses to w, so offset is ln(w) at any prevalence — you never need the true pi.
• Balanced-trained “0.90” really means 0.083 at pi=0.01; feed it to expected-loss or a 0.5 threshold and every number is off by an order of magnitude.

Focal loss

• FL = -alpha_t·(1-p_t)^gamma·log(p_t); p_t = prob assigned to the correct answer. gamma=0 is log loss, gamma=2 standard.
• Down-weights easy examples (p_t=0.9 counts 100x less at gamma=2). Assumes majority rows are easy, not just numerous — built for dense object detection, rarely helps tabular 1:99.
• Not a proper scoring rule → output is a score to rank/threshold, not a probability. Calibrate after if you need probabilities.

Calibration

• Calibrated: among rows scored q, a fraction q are positive. Weak alone — a constant base-rate predictor is perfectly calibrated and useless.
• Brier = reliability − resolution + uncertainty. Fix reliability (small), keep resolution large. Always report a discrimination number (AUC/PR-AUC) beside calibration; a monotone recalibration fixes the first, cannot touch the second.
• ECE = row-weighted mean of |accuracy − confidence| per bin. It is a lower bound (opposite-sign within-bin errors cancel), monotone in bin coarseness, gameable, not proper — always report bin count and mode plus a proper score. When error runs one direction, ECE = |mean conf − accuracy|.
• Calibrators are monotone (never reorder → AUC safe) and must be fit on held-out data (training predictions overstate confidence). Cross-fit if data is scarce.

• Deep nets overconfident: at zero training error NLL still >0, and scaling all logits by k>1 keeps lowering it → no finite minimizer, softmax saturates. Fix: temperature scaling (z/T, monotone, accuracy unchanged).
• Boosted trees / forests under-confident at extremes: leaf values shrunk by eta and lambda, and the step collapses as p→1, so a GBDT may never exceed ~0.96 (a p>0.99 rule silently never fires); forests average correlated votes and compress borderline rows. Fix: Platt (sigmoid-shaped curve) or read as rank.
• Calibration matters only when the number is read at face value: cost-matrix threshold, expected value, pricing/reserving/bid-shading, combining models, abstain thresholds, shown to a human. Not needed for ranked queues, top-k, quantile cutoffs. Argmax needs no calibration but does owe the per-class prior correction.

Drift

Factor p(x,y) two ways; each drift type is one piece moving while its partner holds.

• Covariate shift only hurts via misspecification or extrapolation; high PSI is a prompt to check, not to retrain.
  • Domain classifier d(x)=P(new period) gives density ratio = [d/(1-d)]·(n_old/n_new), the drift magnitude (its held-out AUC: 0.70 look, 0.85 act), and the culprit features — one model, three jobs.
• Label shift: same correction as resampling. Estimate pi_new with BBSE from predictions only: q = TPR·pi + FPR·(1-pi), solve for pi. Sanity check: PSI low but predicted-positive rate moved = genuine label shift.
• Concept drift is the residual (everything visible stable, model wrong). No detector recovers it because y is missing — build a fast proxy label instead.

PSI and monitoring

• PSI = sum (a_i - e_i)·ln(a_i/e_i) over bins frozen at training deciles (e_i=0.10). It is symmetric KL (Jeffreys): KL(a‖e)+KL(e‖a).
• Thresholds <0.10 / 0.10–0.25 / >0.25 are folklore: dominated by emptied bins (epsilon floor sets your alert), and ignore sample size — noise floor E[PSI] ≈ (K-1)/n (fires at “investigate” on ~100-row slices). Bootstrap the floor at your batch size, set alert at p99.
• PSI is univariate (misses joint/correlation drift → domain classifier) and blind to importance (weight by permutation importance).
• Alert ladder — cheapest/fastest first, and pipeline checks fire before statistics:

PIPELINE  schema · nulls · ranges     minutes   absolute → page
INPUT     feature PSI · domain clf     hours     statistical
OUTPUT    pred-positive rate · scores  hours
ACTION    block/approval PER SEGMENT   hours
PROXY     fast label for slow one      days
TRUE      log loss · AUC on labels     weeks     AUDIT, never a page

• Aggregate accuracy is an audit metric, not an alert: late (label latency), diluted (a 5%-segment collapse moves the headline ~2 pts), underpowered (~6,500 labels to see a 1-pt drop).
• Feedback loops hide labels the model’s own decisions suppress (p(x) never moves) → fix structurally with randomized hold-back (1–2%), logged propensities, per-segment action monitoring.