Probability

The small core of probability that ML runs on; every formula is safe only inside the assumption it rests on, and naming that assumption is the skill.

Axioms and the moves that do the work

• Three axioms: P(A) >= 0; P(S) = 1; P(A or B) = P(A) + P(B) when disjoint (cannot co-occur).
• Inclusion-exclusion repairs overlap: P(A or B) = P(A) + P(B) - P(A and B). Summing overlapping risks double-counts and can exceed 1.
• Complement, the move you use most: P(at least one) = 1 - P(none). “None” is one product; “at least one” is a messy sum. Powers birthday, multiple-testing, hash collisions.
• not A must live in the same sample space (complement of “at least one match” is “no matches”, not “exactly one”).

Conditioning, chain rule, independence

• Conditional: P(A|B) = P(A and B)/P(B), P(B) > 0. It is renormalization: drop outcomes outside B, rescale to 1.
• Chain rule: P(x_1..x_n) = prod_t P(x_t | x_<t). An identity, assumes nothing; exactly what an autoregressive LM computes, no approximation.
• Independence: P(A and B) = P(A)P(B), i.e. P(A|B) = P(A). The assumption people supply free and never earn (same user, same session, trees on same data).
• Conditional independence is unrelated to independence, both directions. Fork (X <- Z -> Y): dependent, independent given Z. Collider (X -> Z <- Y): independent, dependent given Z (“explaining away”). Filtering on an outcome is a collider = selection bias: correlations among the approved do not carry to applicants.
• i.i.d. = independent and identically distributed: most-stated, least-checked assumption.

Bayes and base rates

• P(H|E) = P(E|H)·P(H) / P(E), P(E) = sum_h P(E|h)P(h) (hypotheses must be exclusive + exhaustive).
• Odds form (memorize this one): posterior odds = prior odds × likelihood ratio; recover p = odds/(1+odds).
• Failure everyone makes: swapping P(E|H) for P(H|E). They divide by different denominators.
• Base rate has enormous leverage. Fixed 99% sensitivity/specificity test:

• Rare-event landmark: a 99/99 test breaks even at 1% prevalence; at 0.1% it gives ~10 false alarms per true one. The fix is raising the base rate (pre-filter), not a better threshold.

Expectation, variance, covariance

• Linearity of expectation, always, no independence: E[aX + bY + c] = aE[X] + bE[Y] + c. Makes indicator-sum counting arguments work under any dependence.
• E[XY] = E[X]E[Y] only if uncorrelated — this is the line that breaks. E[1/X] != 1/E[X]; ratio-of-means is not mean-of-ratios.
• Var(X) = E[X^2] - (E[X])^2; Cov(X,Y) = E[XY] - E[X]E[Y]; Corr = Cov/(sd·sd) in [-1,1].
• Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y); Var(aX) = a^2 Var(X).
• Uncorrelated != independent: X~U(-1,1), Y=X^2 gives Cov = 0 though Y is determined by X. Correlation screens silently drop every non-monotone feature.
• Bagging floor: average of B vars, pairwise corr rho, gives rho·s^2 + (1-rho)s^2/B. More trees never beat rho·s^2; the only lever is decorrelation.
• Total variance: Var(Y) = E[Var(Y|X)] + Var(E[Y|X]) = within-group + between-group. Every variance decomposition is this with a chosen grouping.
• Bias-variance: E[(y-f_hat)^2] = Bias^2 + Var + sigma^2 (total variance with “group” = training draw). sigma^2 is irreducible; squared-error loss only.

The six distributions

Each is a limit or variation of another, carrying a physical assumption that can fail.

Bernoulli(p) --count n--> Binomial(n,p) --n big, p small, np=lambda--> Poisson(lambda)
     |                          \--n big, np big (CLT)--> Normal(mu, sigma^2)
     \--trials to 1st success--> Geometric(p)      Poisson time-between --> Exponential(lambda)

• Memorylessness: P(T>s+t | T>s) = P(T>t), exponential only. Latency hazard rises — so fixed timeout + retry is correct there.
• CLT: sqrt(n)(Xbar - mu)/sigma -> N(0,1) in distribution. Only the standardized mean converges (not the data); needs iid + finite variance (Cauchy breaks it). Says nothing about tails.
• Berry-Esseen: max CDF error <= 0.4748·rho/(sigma^3·sqrt(n)), rho = E|X-mu|^3. ~0.21 at n=30 for Exponential(1) — n>=30 folklore fails in the tail; bootstrap instead.

Log space, entropy, cross-entropy, KL

• float64 underflows to 0 near 77-81 tokens of chained probs; store log-probs. log(ab) = log a + log b. Adding needs log-sum-exp m + log sum exp(x-m), m = max. Softmax must subtract the max (num and denom, or it rescales).
• H(p) = -sum p log p (best code cost); H(p,q) = -sum p log q (wrong code); KL(p||q) = H(p,q) - H(p) >= 0 (surcharge), 0 iff q=p.
• Cross-entropy = MLE: -(1/N) sum log q(y_i|x_i) = E[H(p_hat, q)]; minimizing it = minimizing KL(p_hat||q). Three names for one objective.
• Special cases: Gaussian noise -> MSE; binary -> -[y log p + (1-y) log(1-p)]. Constant-variance MSE fails when spread grows with level (revenue, latency) -> use a log transform or the right NLL.
• Perplexity = exp(cross-entropy); one-hot row = 1/q(correct). 2.0 nats -> 7.39-way guess.
• KL is asymmetric: forward KL(p_data||q) = mode-covering (hedges, MLE default); reverse KL(q||p) = mode-seeking (variational inference, RLHF penalty). Blows up where a>0, b=0. Always state the direction.

Classic problems