Evaluation Metrics

Every metric is a loss you agreed to be judged by, and each loss drags the model toward one summary (functional) of the true spread p(y|x); picking a metric is picking that summary.

The core move: metric → functional

Strip a metric to its per-row loss l(y,p), ask which constant c minimizes E[l(Y,c)] (argmin = the input that minimizes, not the min value). That c is what the model converges to inside every group of rows it cannot tell apart.

• MSE vs MAE disagree on skewed targets (mean vs median); on y=[1,2,2,3,50], MSE picks the mean-predictor, MAE the median-predictor, opposite winners.
• RMSE ≥ MAE always (Jensen); ratio near 1 = uniform errors, ratio > ~2 = a few rows own the squared error (check for unit errors, sentinels, dupes before touching the model).

Regression gotchas

• MAPE penalizes a 2x over-forecast (100%) same as predicting zero, but over-forecast is unbounded → trains in under-forecasting; undefined at y=0, explodes near it. Default to WAPE = sum|y-p|/sum|y| when someone asks for percent error.
• R^2 = 1 - SSE/SST, baseline = mean of the evaluation set. Goes negative on a level shift (a shifted intercept, small errors). Not comparable across datasets/segments: same RMSE gives R^2 0.96 on a high-variance segment, 0.36 on a low-variance one. Compare RMSE/MAE instead.

Classification: the four cells

Every classification metric is a ratio of TP, FP, FN, TN. Precision’s denominator is a row (model’s decisions, mixes classes); recall’s and FPR’s are columns (fixed by data).

• precision = TP/(TP+FP), recall/TPR = TP/(TP+FN), FPR = FP/(FP+TN).
• Adding negatives cannot change recall/FPR (within-class) but lowers precision.
• Accuracy = (TP+TN)/total is prevalence-weighted → useless under imbalance (a null model hits 99.7% on 0.3% fraud). Dies below ~20% prevalence or when the two mistakes cost differently.
• F1 = harmonic mean 2PR/(P+R), sits near the smaller of P/R, ignores TN (why it survives imbalance). F_beta: beta=2 when misses hurt, beta=0.5 when false alarms hurt.
• MCC reads all four cells, -1..+1; barely moves when you dump in easy negatives. Best single number when the negative class matters.
• Multi-class: micro (pool counts, per-row vote, operational) vs macro (per-class average, catches rare-class failures). Report which.

AUC vs proper scores

• ROC-AUC = P(score(random pos) > score(random neg)); 0.5 = chance, 1.0 = perfect. Rank-only → invariant to calibration and to prevalence (both axes within-class).
• PR-AUC / AP spends resolution where you operate under heavy imbalance, but precision contains pi, so it is not comparable across prevalences (downsampling negatives 0.003→0.5 moved PR-AUC 0.31→0.93, model unchanged). Chance baseline = prevalence itself.
• Log loss (-log q) unbounded → punishes confident wrong; Brier ((q-outcome)^2) bounded by 1, outlier-resistant. Both proper (honest q=p is optimal); accuracy/F1 are not.
• Calibration (numbers right) ⟂ discrimination (order right). AUC-vs-log-loss split: ship the model with better ranking and calibrate it (monotone: Platt, isotonic, temperature); a ranking inversion is permanent, a squashed scale is repairable.

Ranking

• Recall@k: coverage of the relevant set, averageable (fixed denominator) → the retrieval-stage metric. P@k ceiling is min(k,#rel)/k, not averageable.
• MRR: 1/rank of first hit; use only when one right answer exists.
• NDCG@k: graded relevance, gain = 2^rel - 1, discount = 1/log2(i+1) (patient user; 1/i = impatient), normalized by ideal ordering → comparable across queries. A top-of-list swap costs ~33x a deep one.
• Offline NDCG disagrees with online CTR via position bias, the feedback loop over never-shown items, and label mismatch. Fix bias with inverse-propensity scoring (1/P(examine|i), high variance deep down), reserve a randomized slot, decide with interleaving/A-B.

Thresholds

• threshold* = C_fp / (C_fp + C_fn). Equal costs → 0.5 (a cost claim, not a probability fact). Prevalence is absent (already inside p).
• Only valid on a calibrated p; else calibrate or sweep the raw score for min measured cost. If capacity binds, ignore it: take top-K, metric becomes precision@K.
• “We optimized F1” secretly declares a cost ratio: the F1-optimal threshold is F1/2, so C_fn/C_fp = (1-t)/t.