Eigenvalues and Matrix Decompositions

In this lesson, we’ll factor matrices into simpler pieces that reveal what they actually do. Three decompositions show up constantly in ML: eigenvalues and eigenvectors, the spectral theorem for symmetric matrices, and the singular value decomposition (SVD). They run from most specific to most general, and each is a step toward the next. By the end you’ll be able to say what each one factors, which matrices it applies to, and why PCA is really the SVD in disguise.

flowchart TD
  E["Eigen-decomposition: square matrices"] --> S["Spectral theorem: symmetric matrices (real eigenvalues, orthogonal eigenvectors)"]
  S --> V["SVD: any m x n matrix"]
  V --> P["PCA: SVD applied to centered data"]

Reading the chain from top to bottom, each box relaxes a restriction on the box above it, and the last box is the payoff we use every day.

Eigenvalues and eigenvectors

Most vectors change both direction and length when multiplied by a matrix A. An eigenvector is a special vector whose direction is preserved; A only scales it. The scale factor is its eigenvalue lambda:

A v = lambda v      (v is nonzero)

Let’s make that concrete before the algebra. Suppose A doubles everything along the x-axis and leaves the y-axis alone. The vector [1, 0] comes back as [2, 0], same direction, so it is an eigenvector with lambda = 2; the vector [0, 1] comes back unchanged, an eigenvector with lambda = 1. So A acts on its eigenvector v as pure stretching by lambda. The eigenvalues come from the characteristic equation det(A - lambda I) = 0, and each eigenvalue’s eigenvectors span the directions it stretches.

|lambda| > 1 stretches, |lambda| < 1 shrinks, negative lambda flips direction.
Repeatedly applying A amplifies the direction of the largest-magnitude eigenvalue. That dominant eigenvalue governs the long-run behavior of iterated maps and the stability of many numerical methods.

The reason the dominant eigenvalue wins is worth holding onto: every application multiplies each eigendirection by its own lambda, so the largest one outgrows the rest. That is the general square-matrix story; it gets cleaner the moment the matrix is symmetric.

Symmetric matrices and the spectral theorem

Symmetric matrices (A = A^T), which include every covariance matrix, are especially well behaved. The spectral theorem says a symmetric matrix has real eigenvalues and a full set of orthogonal (perpendicular) eigenvectors, so it factors as:

A = Q D Q^T

where Q’s columns are the orthonormal eigenvectors and D is diagonal with the eigenvalues. The factorization amounts to three physical moves: A rotates into the eigenvector axes (Q^T), scales along them (D), then rotates back (Q). Nothing bends; it is only rotate, stretch, rotate.

A symmetric matrix is positive definite when all its eigenvalues are positive. That guarantees a unique minimum for the quadratic it defines, which is why it recurs in optimization and in valid covariance matrices. The one restriction left is squareness, and lifting it gives the most general tool of the three.

Singular value decomposition (SVD)

Eigen-decomposition needs a square matrix. The SVD works for any (m x n) matrix and is the more general tool:

A = U S V^T

U (m x m) and V (n x n) are orthogonal, and S (m x n) is diagonal with non-negative singular values sorted largest to smallest. So any matrix at all, square or not, is still just a rotation, a scaling, and another rotation.

flowchart LR
  X["input vector"] --> VT["V^T: rotate input axes"]
  VT --> S["S: stretch by singular values"]
  S --> U["U: rotate to output axes"]
  U --> Y["A x"]

The singular values measure how much A stretches along each principal direction, and the number of nonzero ones is the rank. Because they are sorted largest first, keeping only the top k singular values gives the best rank-k approximation of A, the basis of low-rank compression and denoising: we spend our budget on the directions that stretch the most and drop the rest. That ordering by importance is exactly what makes the SVD the engine behind PCA.

The tie to PCA

Principal Component Analysis diagonalizes the data’s covariance matrix. Because covariance is symmetric and positive semi-definite, its eigenvectors are orthogonal directions of variance and its eigenvalues are the variance along each. The top eigenvectors capture the most variance, so projecting onto them reduces dimensionality with the least information loss.

Running the SVD directly on the centered data matrix yields the same principal directions without ever forming the covariance matrix, which is how PCA is computed in practice (forming the covariance matrix squares the condition number and loses precision). So the practical recipe skips the covariance matrix entirely and goes straight to the SVD, which closes the loop from the diagram we opened with.

Conclusion

An eigenvector keeps its direction under A; its eigenvalue is the scale factor, and the dominant one drives long-run and stability behavior.
The spectral theorem makes symmetric matrices factor cleanly as Q D Q^T with real eigenvalues and orthogonal eigenvectors; positive definite means all eigenvalues are positive.
The SVD generalizes this to any matrix as U S V^T; the top singular values give the best low-rank approximation.
PCA is the eigen-decomposition of the covariance matrix, computed in practice as the SVD of the centered data.

One line to remember: every decomposition here says the same thing at a different level of generality, a matrix is rotate, stretch, rotate, and the biggest stretches are the ones worth keeping.

Eigenvalues and eigenvectors

Symmetric matrices and the spectral theorem

Singular value decomposition (SVD)

The tie to PCA

Conclusion

Further reading