Feature Engineering — cheat sheet — Machine Learning

Turning a raw row into a fixed-length numeric vector; every transform decision falls out of the model’s own math, and the recurring hazard is a statistic fit on rows you will later score.

The pipeline

Raw -> Transform -> Encode -> Impute -> Cross -> Select -> Feature store -> {Training, Serving}

Every box is fit on data (a scaler stores a mean, an imputer a median, a target encoder one stat per category); those numbers ship with the model.
Leakage: a feature carries info unavailable at prediction time. Rule: if any box is fit on rows later scored, you leaked. Symptom: great offline, poor in production.
The fork is where training and serving must run the same function; when they diverge nothing errors and no monitor fires.

Scaling: verdict by model math

Model	Scaling	Why
Linear/logistic + SGD, NN	Required	Condition number `kappa = L/mu` drives convergence
Ridge/Lasso/ElasticNet	Required	Penalty is on the raw coefficient scale
kNN, k-means, SVM-RBF, PCA	Required	Distance sums raw units
Tree, RF, GBDT	Cosmetic	Splits `x <= t` invariant under monotone `f`
OLS (closed-form), Gaussian NB	Not required	No iteration / no cross-feature metric

Standardize (x-mean)/sd: default; breaks when one outlier moves mean and inflates sd. Min-max (x-min)/(max-min): bounded range, but a test value outside [min,max] escapes silently. Robust (x-median)/IQR: heavy tails, order-based, barely breaks.
Unscaled gradient descent example: kappa 9e6 needs ~6.2e7 iterations; standardized kappa=1 lands in 1 step.
Gotcha: a monotone transform of the target is not cosmetic for a regression tree (Var(log y) != Var(y)).

Log transform: three jobs

Multiplicative -> additive: y = a·x1^b·x2^c becomes log y = log a + b·log x1 + c·log x2. Capacity fix, not cosmetic.
Variance stabilization: if sd(y|x) = c·E[y|x], delta method gives Var(log y) ≈ c^2 (constant). Removes heteroscedasticity.
Killing leverage: h_i can hit 0.99 for one outlier; log drops it to ~0.12.
Pure ceremony on tree ensembles (monotone-invariant). Box-Cox needs x>0; Yeo-Johnson tolerates zeros/negatives; both fit lambda you must persist.

Categorical encoding

Route by cardinality k:

k <= ~15: one-hot (k-1 dummies). Cost is statistical: Var(beta_j) = sigma^2/n_j, so 2 rows/category gives noise.
k ~15..1000: one-hot if 100+ rows each; else target encoding + smoothing.
k > 1000: hashing (linear/streaming), embedding (NN), native categorical (GBDT).
Target encoding: enc(c) = mean(y|c). A singleton category’s encoding is y_i -> leaks the label. Fix: out-of-fold (OOF) so row i’s encoding never saw y[i], plus smoothing (n_c·mean_c + m·global)/(n_c + m). The prior must come from the training fold only.
Leakage risk: one-hot / ordinal / hashing / count = none; target = high; embedding = low.
Ordinal encoding invents a false metric (equal spacing); reserve for genuinely ordered levels.

Missing values: mechanism first

Mechanism	Absence depends on	Response
MCAR	nothing	any imputation unbiased; deletion valid but wasteful
MAR	other observed columns	conditional imputation (MICE) + indicator
MNAR	the missing value itself	no imputation recovers it; model the missingness

Mean imputation damage: Var = (1-f)·Var_true; reported SE shrinks to (1-f) of truth, so at f=0.3 intervals are 30% too narrow, t-stats 43% too large.
Even under MCAR, mean imputation biases coefficients once predictors are correlated (Cov(x*,·) shrinks by (1-f), others don’t).
Universal move: add an x_is_missing indicator, always. GBDT: pass NaN, it learns a default split direction. Categorical: "MISSING" as its own level.

Temporal leakage & train/serving skew

Rule: every feature value must be computable from data timestamped before t and available at t (two conditions).
Trap 1 unbounded aggregate: add WHERE ts < prediction_time to every query. Trap 2 rolling window: .shift(1) so today’s value isn’t in today’s mean. Trap 3 restated data: needs bitemporal storage (valid_time + transaction_time). Trap 4 split: use forward-chaining CV with a purge gap = label horizon.
Cyclical time: encode hour as sin(2·pi·h/24), cos(...) so 23:00 and 00:00 are adjacent.
Training/serving skew = same feature name, different code across the offline/online fork. Drift monitors miss it (identically wrong since launch). Prevent (log-and-train, one definition/one engine) beats detect (skew alert on p99 diff, per-slice metrics).

Feature selection

Filter (O(p), no fit) misses interactions: for XOR, MI(x1;y)=0 exactly while the pair gives 1 bit. Embedded (L1, tree gain) = one fit, inherits model bias. Wrapper (RFE) = O(p·k) fits, expensive.
The classic fake result: selecting on the full dataset then cross-validating. With p=5,000 noise columns and n=100, best |r| ≈ 0.39 by chance. Selection is model fitting: put it inside the CV loop.
Order: drop constants/unservable -> L1 or tree shortlist (5,000->~200) -> permutation importance on held-out -> wrapper on the final shortlist only.