Pseudoinverses, Low Rank and Conditioning

Invert only the represented directions

Given $A=U\Sigma V^T$ , form $\Sigma^+$ by transposing its shape and replacing each positive singular value by its reciprocal, leaving zeros at zero. The Moore–Penrose pseudoinverse is $A^+=V\Sigma^+U^T$ .

Then $x_*=A^+b$ is the least-squares solution with smallest Euclidean norm. If the system is consistent, it is the minimum-norm exact solution. It does not manufacture an exact solution when the target is outside the column space.

Visual guide

VISUAL GUIDEChoose the shortest solution among many

All exact solutions

For x + y = 2, every point on the blue line is a solution. The pseudoinverse selects (1, 1), the point nearest the origin. The displacement to it is perpendicular to the nullspace direction (1, −1).

Worked example: an inconsistent target

For $A=\operatorname{diag}(2,0)$ , the pseudoinverse is $\operatorname{diag}(1/2,0)$ . With $b=(6,4)^T$ , it gives $x_*=(3,0)^T$ , fitted vector $(6,0)^T$ , and residual $(0,4)^T$ . Any vector $(3,t)^T$ produces the same fit, but $t=0$ has the smallest norm.

Worked example: noise amplification

For $A=\operatorname{diag}(1,0.001)$ , solving $Ax=b$ multiplies the second component of $b$ by $1000$ . A measurement error of $0.001$ in that component creates an error of $1$ in the solution. For invertible matrices, the spectral condition number is $\kappa_2(A)=\sigma_{\max}/\sigma_{\min}$ , here $1000$ .

Conditioning describes sensitivity of the mathematical problem. Stability describes the algorithm's additional error. A stable algorithm cannot eliminate sensitivity already present in the model.

Low-rank approximation

The expansion $A=\sum_{i=1}^r\sigma_i u_iv_i^T$ separates rank-one contributions. Keeping the first $k$ terms gives a best rank-at-most- $k$ approximation in spectral and Frobenius norms. The spectral error is $\sigma_{k+1}$ ; the squared Frobenius error is $\sum_{i>k}\sigma_i^2$ .

Discarding small singular directions can reduce noise amplification, but changes the problem and introduces approximation bias. A cutoff should reflect measurement scale or a stated tolerance, not an unexplained universal threshold.

Practice

Find the pseudoinverse of $\operatorname{diag}(4,0)$ .
Find the condition number of $\operatorname{diag}(2,5)$ .
Singular values are $10,3,0.2$ . What is the best rank-two spectral error?

Show worked solutions

$\operatorname{diag}(1/4,0)$ .
$5/2$ .
The discarded singular value is $0.2$ .