Singular Value Decomposition

A factorization beyond eigenvectors

Every real $m\times n$ matrix has an SVD $A=U\Sigma V^T$ . Here $U$ and $V$ are square orthogonal matrices and $\Sigma$ is rectangular diagonal with nonnegative entries $\sigma_1\ge\cdots\ge\sigma_r>0$ , followed by zeros. The positive singular values count the rank.

The right singular vectors $v_i$ are input directions, and the left singular vectors $u_i$ are output directions, satisfying $Av_i=\sigma_i u_i$ . Read the factorization as orthogonal coordinate conversion, axis stretching or collapse, and another orthogonal transformation. Reflections may occur as well as rotations.

Visual guide

VISUAL GUIDEA circle becomes an ellipse

Unit input circle
Image ellipse

For A = diag(3, 1), unit input directions become an ellipse with semi-axis lengths 3 and 1. These are the singular values. General SVD adds rotations before and after this axis-aligned stretching.

Worked example: different input and output sizes

For $A=\begin{pmatrix}3&0\\0&2\\0&0\end{pmatrix}$ , choose $V=I_2$ , $U=I_3$ , and $\Sigma=A$ . The unit circle in the input plane becomes an ellipse in the output $xy$ -plane, with semiaxes $3$ and $2$ . Its column space has dimension two inside a three-dimensional output space.

Worked example: singular values are not signed eigenvalues

For $A=\operatorname{diag}(-3,2)$ , eigenvalues are $-3,2$ , but singular values are $3,2$ . The sign reversal can be carried by an orthogonal factor. Singular values measure nonnegative stretch, not orientation.

The eigenvalues of $A^TA$ are $\sigma_i^2$ , and its orthonormal eigenvectors give the right singular directions. For each positive $\sigma_i$ , obtain $u_i=Av_i/\sigma_i$ . Zero singular values require completing an orthonormal basis rather than dividing by zero.

SVD exists even when a matrix is rectangular, singular or not diagonalizable. Its factors are not always unique: singular-vector signs can be paired differently, and repeated singular values allow rotations within their associated subspaces.

Practice

Find the singular values of $\operatorname{diag}(4,-1)$ .
If a $5\times3$ matrix has two positive singular values, find its rank and nullity.
What is the largest possible $\|Ax\|$ for $\|x\|=1$ ?

Show worked solutions

$4$ and $1$ .
Rank two and input nullity $3-2=1$ .
$\sigma_1$ , attained by a leading right singular vector.