Classical Models

Six supervised families for tabular data, each an objective plus a bet: name the assumption you are betting on and you name the failure mode in the same breath.

Picking a family

On tabular data gradient boosting is the default; everything else needs a reason (small n, interpretability, latency, calibrated extrapolation).

flowchart TD
    S{"Tabular?"} -->|Text| TXT["Linear SVM / logistic on TF-IDF"]
    S -->|Images| IMG["Pretrained backbone + linear head"]
    S -->|Yes| N{"n rows"}
    N -->|"< ~1k"| P{"Need a probability?"}
    N -->|"1k–10M"| GBDT["Gradient boosting"]
    N -->|"> ~10M"| LIN["Linear / logistic + SGD"]
    P -->|Yes| LR["Regularized logistic"]
    P -->|No| SVM["Kernel SVM"]
    GBDT -.->|"must explain coefficients"| LR
    style GBDT fill:#2d6a4f,color:#fff

Linear regression

• OLS: minimize ||y - Xw||^2; convex, closed form w = (X^T X)^-1 X^T y (normal equations). Only model here with no iteration.
• Cost: closed form O(np^2 + p^3), memory p^2 (Gram matrix); SGD O(np)/epoch, memory p, streams. p (not n) sets the crossover.
• Never compute inv(X.T @ X): forming X^T X squares the condition number, so kappa^2 · eps blows up. Use QR/SVD (lstsq).
• Only two assumption violations bias the coefficients: non-linearity and exogeneity (E[e|X]=0). Non-independence, heteroscedasticity, non-normal errors break only the standard errors (fix with HC3 / Newey-West).
• Collinearity: Var(beta_j) ∝ 1/(1 - R_j^2); VIF = 1/(1-R_j^2), SE inflates by sqrt(VIF). Hurts interpretation only, never prediction.

Regularization: L1 vs L2

• Ridge (L2) penalizes sum w^2; Lasso (L1) penalizes sum |w|. lambda=0 is OLS. Standardize first (penalty acts on raw units); never penalize the intercept; tune lambda by CV.
• One coordinate (orthonormal design), OLS value z:

• Why L1 zeroes: the subgradient of |w| at 0 is the whole interval [-λ, λ], so 0 is optimal for a range of gradients. L2’s gradient 2λw vanishes at 0, so it never reaches zero. (Geometry: diamond corners on the axes vs smooth ball.)
• Rules: ridge w = z/(1+2λ); lasso w = sign(z)·max(|z|-λ, 0).
• Elastic net for correlated feature blocks: L2 term makes the objective strictly convex, so the non-unique lasso pick (a coin flip) becomes a stable shared weight (grouping effect).

Logistic regression

• Squash score w·x with the sigmoid 1/(1+e^-z), fit by max likelihood. Boundary is a flat hyperplane at w·x=0. Logit = log-odds is the canonical link for Bernoulli.
• Not MSE: over a sigmoid MSE is concave for s < 1/3 (non-convex) and its gradient is ~50× smaller at s=0.01 (vanishing). Log loss is convex everywhere with clean gradient s - y.
• beta_j = change in log-odds; exp(beta_j) = odds ratio (constant). The probability change is not constant: OR 2 moves p 0.10→0.18 but 0.50→0.67.
• Complete separation sends the MLE to infinity (coef=24.7 + convergence warning); L2 (sklearn C=1.0, C = inverse penalty) fixes it.

Trees, kNN, SVM, naive Bayes

• Trees: recursive x <= t splits, greedy max gain (Gini 1 - Σp² or entropy). Two overfit paths: (1) max gain over ~p·n candidates is positive even with zero signal (>4 sd at 50k candidates); (2) rows halve each level, leaves become single memorized rows. Controls: min_samples_leaf (best single knob), ccp_alpha post-pruning, max_depth. No scaling needed, no extrapolation (leaf means flatline outside training range → trending features fail silently).
• kNN: no training, O(nd)/query. Curse of dimensionality: relative distance spread = 1.183/sqrt(d), so past ~20–30 features “nearest” stops meaning anything. Works on embeddings because intrinsic dimension (10–30) drives it, not ambient. Standardize, reduce dimension, pick k by CV, distance-weight.
• SVM: maximize margin 2/||w||; soft margin adds slack priced at C (inverse regularization). Hinge loss is exactly 0 past the margin, so only support vectors matter (a million easy points don’t move the boundary). Kernel trick: dual uses only inner products, so replace with K(x,z) (RBF = infinite-dim map). Died on big tabular because the kernel matrix is n×n (80 GB at n=100k). Still wins when p > n. Outputs distance not probability → Platt-scale.
• Naive Bayes: P(y|x) ∝ P(y)·Π P(x_j|y), assumes conditional independence (false everywhere). Survives because correlated evidence distorts the probability but not the argmax/ranking. Laplace smoothing (α=1) is mandatory or one unseen token zeroes the product. Use as label/rank only; isotonic-calibrate before thresholding. O(n·nnz), streams.

Symptom → fix