Least Squares and Data Fitting

2 MIN READ · PRACTISE AT YOUR OWN PACE · FREE

Replace exact matching with closest matching

When $Ax=b$ is inconsistent, least squares seeks $\hat x$ minimizing $\|Ax-b\|^2$ . The fitted vector $A\hat x$ is the orthogonal projection of $b$ onto $C(A)$ , so the residual $r=b-A\hat x$ is perpendicular to every column of $A$ . Therefore

A^TA\hat x=A^Tb.

These normal equations always have a solution. The coefficient vector is unique when $A$ has independent columns. Otherwise the fitted vector is still unique, but several coefficient vectors may produce it.

Visual guide

VISUAL GUIDEFit by minimizing squared vertical residuals

Least-squares line
Residual

The least-squares line for (0, 1), (1, 2), (2, 2) is y = 7/6 + x/2. The vertical segments are residuals; their squares, not their signed sum alone, define the minimized error.

Worked example: fit a line

Fit $y=\alpha+\beta t$ to $(t,y)=(0,1),(1,2),(2,2)$ . Then

A=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix},\quad b=\begin{pmatrix}1\\2\\2\end{pmatrix},\quad A^TA=\begin{pmatrix}3&3\\3&5\end{pmatrix},\quad A^Tb=\begin{pmatrix}5\\6\end{pmatrix}.

Solving gives $\beta=1/2$ , $\alpha=7/6$ . The residual is $(-1/6,1/3,-1/6)^T$ . Its dot product with the constant column is zero, and with the time column is also zero. This verifies the least-squares condition.

PAUSE & THINKA quick check, not a grade

Try it yourself.

Choose an answer and explain your reasoning to yourself. Use a hint if you get stuck.

At a least-squares solution x̂, how is r=b−Ax̂ related to the columns of A?

It must equal zero.

It is perpendicular to every column.

It is parallel to every column.

Hint 1 · Find a starting point

A residual that still points along a column could be reduced.

Hint 2 · Take the next step

The normal equations give Aᵀr=0.

Show the reasoning

Answer: It is perpendicular to every column.

The optimal residual is orthogonal to the column space, even when an exact solution is impossible.

Before moving on: what would make one of the other answers wrong? Saying why is part of understanding.

Worked example: repeated measurements

For a constant model $y=c$ fitted to measurements $2,4,9$ , the matrix is a column of ones. The normal equation is $3c=15$ , so $c=5$ , the arithmetic mean. Residuals $-3,-1,4$ sum to zero.

Ordinary least squares treats all residuals with equal weight and squares their magnitudes, making large errors influential. Measurement uncertainty may motivate weighted least squares, while outliers may call for a different model. A mathematically correct fit does not establish that the physical relationship is truly linear.

For numerical work, solve through QR or SVD. Forming $A^TA$ squares the spectral condition number for a full-rank matrix and can lose accuracy when columns are nearly dependent.

Practice

Fit a constant to $3,6,6$ .
If $b$ already lies in $C(A)$ , what is the minimum residual?
Why can a fitted vector be unique when the coefficients are not?

Show worked solutions

The mean is $5$ .
Zero: an exact solution is available.
Adding any vector in $N(A)$ to the coefficients leaves $A\hat x$ unchanged.

MAKE IT YOURS

Pause before the next idea.

Can you explain how the visualization connects to this lesson’s goal? If a step still feels uncertain, put this lesson on your review list and try the check again another day.

Optional marks, not a grade. Saved in this browser only. Open notebook →