Orthogonality and Least Squares
Cheat sheet.

The key rules, formulas and reminders from all 3 topics, gathered into reference cards.

Key formulas, conditions and traps · Read down each column.

Dot Products and Orthogonal Projection

p=\frac{a^Tb}{a^Ta}a,\quad r=b-p,\quad a^Tr=0,\quad \|b\|^2=\|p\|^2+\|r\|^2.

The direction vector must be nonzero. Projection onto a line is a vector, not just its scalar coefficient.

Subtract earlier projections, then normalize. For full column rank, $A=QR$ with $Q^TQ=I$ and upper triangular invertible $R$ .

Zero residuals indicate dependence. Rectangular $Q$ does not satisfy $QQ^T=I$ on the entire output space.

\min_x\|Ax-b\|^2\quad\Longrightarrow\quad A^T(b-A\hat x)=0.

Full column rank gives unique coefficients. QR or SVD is generally safer numerically than forming normal equations.

2 reference blocks

p=\frac{a^Tb}{a^Ta}a,\quad r=b-p,\quad a^Tr=0,\quad \|b\|^2=\|p\|^2+\|r\|^2.

The direction vector must be nonzero. Projection onto a line is a vector, not just its scalar coefficient.

2 reference blocks

Subtract earlier projections, then normalize. For full column rank, $A=QR$ with $Q^TQ=I$ and upper triangular invertible $R$ .

Zero residuals indicate dependence. Rectangular $Q$ does not satisfy $QQ^T=I$ on the entire output space.

2 reference blocks

\min_x\|Ax-b\|^2\quad\Longrightarrow\quad A^T(b-A\hat x)=0.

Full column rank gives unique coefficients. QR or SVD is generally safer numerically than forming normal equations.