Eigenvalues and Matrix Decompositions

Every decomposition here says the same thing at a different level of generality: a matrix is rotate, stretch, rotate, and the biggest stretches are worth keeping.

Eigenvalues and eigenvectors

• Eigenvector v: direction preserved under A; A only scales it. Eigenvalue lambda: the scale factor.
• A v = lambda v (v nonzero); eigenvalues solve det(A - lambda I) = 0.
• |lambda| > 1 stretches, |lambda| < 1 shrinks, lambda < 0 flips direction.
• Iterating A amplifies the largest-magnitude (dominant) eigenvalue; it governs long-run behavior and numerical stability.

Spectral theorem (symmetric matrices)

• Applies to symmetric A = A^T (includes every covariance matrix).
• Guarantees real eigenvalues and a full set of orthogonal eigenvectors.
• Factors as A = Q D Q^T: Q columns are orthonormal eigenvectors, D diagonal of eigenvalues.
• Three moves: rotate into eigen-axes (Q^T), scale (D), rotate back (Q).
• Positive definite = all eigenvalues positive; gives a unique minimum for its quadratic.

Singular value decomposition (SVD)

• Works for any (m x n) matrix; the most general tool.
• A = U S V^T: U (m x m) and V (n x n) orthogonal; S (m x n) diagonal of non-negative singular values, sorted largest first.
• Singular values = stretch along each principal direction; count of nonzero ones = rank.
• Keeping top k singular values gives the best rank-k approximation (compression, denoising).

x --> [V^T: rotate] --> [S: stretch] --> [U: rotate] --> A x

PCA tie-in

• PCA diagonalizes the data’s covariance matrix (symmetric, positive semi-definite).
• Eigenvectors = orthogonal directions of variance; eigenvalues = variance along each; top ones capture most variance.
• In practice: run SVD on the centered data matrix directly, same principal directions.
• Forming the covariance matrix squares the condition number and loses precision, so skip it.

Which decomposition applies