Statistics and Inference

Statistics runs probability backwards: from one sample, recover a best guess (point estimate), an honest interval, and a ship/no-ship decision with a stated error rate, and every procedure silently trusts an assumption.

Estimators: judge by MSE

• Estimator: a recipe turning a sample into a number; being random, it has a sampling distribution. Every property below is a property of that distribution.
• Bias = E[theta_hat] - theta; Variance = spread across samples; MSE = bias^2 + Var is the only thing that matters when choosing between estimators.
• Consistency: theta_hat -> theta as n -> infinity. Independent of unbiasedness: “always report X_1” is unbiased but never converges; the MLE variance is consistent but biased at every n.
• Bias can pay: with mu=10, sigma=10, n=4, shrinking to 0.8·xbar scores MSE 20 vs 25 for plain xbar. Optimal c* = mu^2/(mu^2+sigma^2/n) needs the answer, so nobody uses it.

      one estimator (random variable)
        /                    \
     Bias                 Variance
        \                    /
         MSE = bias^2 + Var       <- optimize this, not bias alone

Maximum likelihood and the divisor

• MLE: pick the parameter that makes the observed data most probable. Recipe: write likelihood, take log, differentiate, set to 0, check 2nd derivative < 0.
• Bernoulli: p_hat = k/n. Gaussian: mu_hat = xbar, sigma^2_hat = (1/n)·sum(x_i - xbar)^2.
• MLE variance is biased low: E[sigma^2_MLE] = ((n-1)/n)·sigma^2, because xbar minimizes sum(x_i - c)^2 by construction, understating spread around the true mu by one degree of freedom.
• Divisor is a choice of loss: /n = MLE, /(n-1) = unbiased, /(n+1) = minimum-MSE. On [2,4,4,4,6] (SS=8): 1.60, 2.00, 1.33.
• Jensen catch: s^2 is unbiased for sigma^2, but E[s] < sigma (square root is concave). Unbiasedness dies under nonlinear transforms.
• Assumes correct model, independence, identical distribution, interior smooth maximum. The n-1 fix is exact only under independence.

Intervals and p-values

• CI for a mean: xbar +/- t_{0.975,n-1}·s/sqrt(n). SE = s/sqrt(n) is the spread of the estimate, not of rows; SE shrinks with n, s does not.
• Correct reading: the procedure covers the truth 95% of the time. Wrong: “95% chance the truth is in this interval” (frequentist parameter is fixed, not random).
• PI (next single observation): xbar +/- t·s·sqrt(1+1/n), ~10x wider at n=100. Stakeholders asking “will a user convert 50-54%?” mean the PI; handing them the CI drives over-confident forecasts.
• 95% CI = null values a two-sided alpha=0.05 test fails to reject (same pivot only): excludes zero <-> p < 0.05.
• p-value = P(data-or-worse | H0, pre-committed stopping rule). NOT P(H0 | data) (that needs a prior), not “probability of chance”, not replication probability.
• Significance != truth: prior 0.10 + power 0.80 -> 36% of “wins” are false; drop power to 0.50 -> 47%. Underpowered studies make discoveries less likely to be real.

Errors and A/B sizing

• Four cells: Type I (false alarm, rate alpha, you set it); Type II (miss, rate beta); power = 1-beta. “Fail to reject” is not “accept”.
• MDE: the smallest effect worth detecting; you cannot pin beta without it.
• Sample size per arm: n = 2·sigma^2·(z_a + z_b)^2 / delta^2 ≈ 16·sigma^2/delta^2 (planning docs use 7.84888, not 16). For a proportion swap 2·sigma^2 -> p1·q1 + p2·q2; using the smaller variance twice quietly delivers 76% power, not 80%.
• Scaling law: n ~ 1/delta^2, so halving the detectable effect quadruples the traffic.

A/B traps (nothing errors out)

• Randomization unit sets what n means. Randomize by user (UI changes), session (invisible backend), cluster (interference). Analyze at a finer level and the design effect 1 + (k-1)·rho_intra inflates n; understated SE turned p=0.03 into p=0.26.
• Peeking: fixed-n broken. 5 looks -> 14.2% Type I; continuous monitoring hits p<0.05 with probability 1 eventually. Fix with group-sequential / mSPRT / confidence sequences (10-25% more sample).
• Multiple testing: FWER = 1 - 0.95^m = 64% at m=20. Bonferroni (alpha/m, any dependence) costs 1.9x sample at m=20; BH controls the false-discovery fraction for screening.
• Novelty vs primacy: effect decays vs grows; plot by days-since-exposure, not calendar day. Interference: treatment steals control’s supply; fix with cluster/switchback (structural, not statistical).
• CUPED: Y - theta·(X - E[X]), theta* = Cov(X,Y)/Var(X), cuts variance to Var(Y)(1-rho^2). rho=0.6 -> 1.56x effective n. X must be strictly pre-randomization (else it is a mediator).
• Bootstrap: resample the randomization unit with replacement, recompute the statistic, use the spread. Fails silently on max/min, dependent rows, tiny n.
• Simpson’s paradox: a treatment can win every subgroup and lose overall (weighted averages, different weights). Adjust for a confounder, never a mediator or collider — a causal (DAG) claim no test can make for you.