Least squares regression and residualsEdexcel A-Level Further Maths: Revision notes

The least squares regression line of $y$ on $x$ is the line $y=a+bx$ that minimises the sum of the squares of the residuals, the vertical distances from the points to the line. $y$ is the response variable and $x$ the explanatory variable, and the line should be used to predict $y$ from $x$ (not the other way round). The derivation is not required; you must use the formulae. $S_{xx}=\sum x^2-\frac{(\sum x)^2}{n},\quad S_{yy}=\sum y^2-\frac{(\sum y)^2}{n},\quad S_{xy}=\sum xy-\frac{\sum x\sum y}{n}.$

$b=\frac{S_{xy}}{S_{xx}},\qquad a=\bar y-b\bar x.$ The line always passes through $(\bar x,\bar y)$. Worked example: $n=6$, $\sum x=114$, $\sum y=342$, $\sum x^2=2236$, $\sum xy=6750$. Then $S_{xx}=2236-\frac{114^2}{6}=70$ and $S_{xy}=6750-\frac{114\times342}{6}=252$, so $b=3.6$. With $\bar x=19$ and $\bar y=57$, $a=57-3.6\times19=-11.4$, so $y=-11.4+3.6x$. The gradient $b$ is the average change in $y$ for each unit increase in $x$; the intercept $a$ is the predicted $y$ when $x=0$, which may not be meaningful.

A residual is the difference between an observed value and the value predicted by the line: $\text{residual}=y_i-\hat y_i=y_i-(a+bx_i).$ A positive residual means the point is above the line; a negative residual means it is below. Residuals from a least squares line always sum to zero. Example: for $y=3+2.2x$, the observation $(6,15)$ has $\hat y=16.2$, so the residual is $15-16.2=-1.2$.

The residual sum of squares is the quantity that least squares minimises: $\text{RSS}=\sum(y_i-\hat y_i)^2=S_{yy}-\frac{(S_{xy})^2}{S_{xx}}.$ A small RSS relative to $S_{yy}$ means the line fits well. Example: $S_{yy}=920$, $S_{xy}=252$, $S_{xx}=70$ gives $\text{RSS}=920-907.2=12.8$. Because it is a sum of squares, RSS is never negative; the formula's second term can never exceed $S_{yy}$.

Use residuals to judge whether a linear model is reasonable. If they are small and show no pattern (a mixture of positive and negative), a linear fit is sensible. A run of positive then negative then positive residuals suggests curvature, so a different model may be needed. One residual much larger than the rest marks a possible outlier. To refine a model, investigate the outlier. If there is a reason (a misreading, a recording error), remove it and recalculate the line and RSS. Example: removing a point with residual $6$ from seven points cut the RSS from $46$ to $4$. Do not remove a point only because it spoils the fit.

Predicting $y$ for an $x$ inside the data range is interpolation and is usually reliable. Predicting outside the range is extrapolation and may be wildly wrong, because the relationship may not stay linear. Always interpret the gradient in context: 'for each extra hour of revision the score increases by $4.3$ marks, on average'. Check that predictions make sense (for example, a test score over the maximum is impossible).