A fun way to add a line of best fit to a dataset using the Least-Squares method. This project is to supplement my OWN LEARNING.

The parameters of a line of best fit is calculated using the normal equations. Given a matrix equation $A\vec{x} = \vec{b}$, the normal equation is that which minimises the sum of the square differences:

$A^TA\hat{\vec{x}} = A^T\vec{b} \;\;\;\Rightarrow \;\;\;\hat{\vec{x}} = (A^TA)^{-1}A^T\vec{b}$

This can be used with the following matrix equations of the form $A\vec{x} = \vec{b}$ to obtain the parameters to an equation that will minimise the cost.

There exists some $\hat{\vec{x}}$ s.t. $A\hat{\vec{x}}$ is the best approximation of $\vec{b}$ in the column space of the $m \times n$ matrix $A$. By the best approximation theorem, the best approximation of $\vec{b}$ in $Col(A)$ is:

$\hat{\vec{b}} = proj_{Col(A)}(\vec{b})$

Any least squares solution must thus satisfy the equation $A\hat{\vec{x}} = proj_{Col(A)}(\vec{b})$. This equation must be consistent, since $\hat{\vec{b}}$ is an element of the column space of A.

We know that $\vec{b} - \hat{\vec{b}}$ must be orthogonal to the column space of A, hence

where $a_i$ represents the columns of $A$ for all $i = 1,2,3,\ldots,n$. Thus: