Least squares regression and residuals
Studying this course? Keep your progress in one place.
Key facts
- The least squares regression line minimises the sum of the squares of the residuals.
- and ; the line passes through .
- Residual observed predicted; residuals sum to zero.
- : smaller means a better fit.
- Interpolation is reliable; extrapolation is not.
The least squares line
It is the line that minimises the sum of the squared vertical distances from the points.
The least squares regression line of on is the line that minimises the sum of the squares of the residuals, the vertical distances from the points to the line.
is the response variable and the explanatory variable, and the line should be used to predict from , not the other way round. To predict from you need a different line.
The derivation is not required; you must use the formulae for , and .
You have the regression line of on . You want to predict from a given . What should you do?
Calculating the coefficients
Find , then .
The gradient is and the intercept is . The line always passes through .
The gradient is the average change in for each unit increase in . The intercept is the predicted when , which may not be meaningful.
Compute and separately first and write them down; they also feed the RSS.
- Gradient
- Intercept
Worked example
, , , , . Find the regression line of on .
- 1
.
- 2
.
- 3
.
- 4
and , so .
, , , . What is ?
Residuals
A residual is observed minus predicted.
A residual is the difference between an observed value and the predicted value from the line: .
A positive residual means the point is above the line; a negative residual means it is below.
Residuals from a least squares line always sum to zero. Subtracting the other way round (predicted minus observed) is a common mistake.
- Residual
Worked example
For , find the residual for the observation .
- 1
Predicted value: .
- 2
Residual .
An observed value is 20 and the line predicts 17. What is the residual?
Residual sum of squares
The RSS is the quantity least squares minimises; a small RSS means a good fit.
The residual sum of squares is .
A small RSS relative to means the line fits well.
Because it is a sum of squares, RSS is never negative; the second term of the formula can never exceed .
- Residual sum of squares
Worked example
, , . Find the RSS.
- 1
.
- 2
.
For the same , which RSS suggests the better fit?
Checking the fit and outliers
Small residuals with no pattern suggest a linear fit; a pattern suggests curvature, and one large residual an outlier.
Use residuals to judge whether a linear model is reasonable.
- Small residuals with no pattern (a mixture of positive and negative): a linear fit is sensible.
- A run of positive, then negative, then positive residuals suggests curvature, so a different model may be needed.
- One residual much larger than the rest marks a possible outlier.
To refine a model, investigate the outlier. If there is a reason (a misreading, a recording error), remove it and recalculate the line and RSS. Do not remove a point only because it spoils the fit. In a comment, name the residual and the change in RSS.
- 1
Spot it
one residual much larger than the rest
- 2
Investigate
look for a misreading or recording error
- 3
Remove only if justified
not just because it spoils the fit
- 4
Recalculate
the line and the RSS
A residual plot shows positive, then negative, then positive residuals in a clear pattern. What does this suggest?
Using and interpreting the line
Predictions inside the data range are usually reliable; outside it they may be wildly wrong.
Predicting for an inside the data range is interpolation and is usually reliable.
Predicting outside the range is extrapolation and may be wildly wrong, because the relationship may not stay linear.
Always interpret the gradient in context: "for each extra hour of revision the score increases by marks, on average". Check that predictions make sense, for example a test score over the maximum is impossible.
Interpolation
- x inside the data range
- Usually reliable
Extrapolation
- x outside the data range
- May be wildly wrong
Data on revision hours ranges from 1 to 10 hours. Which prediction is the least reliable?
Try an exam question
A set of points has , , and . Find the equation of the regression line of on .
[5 marks]
- [1].
- [1].
- [1].
- [1].
- [1].
That's the notes covered.
Carry on to the next subtopic.
Try a question
Find the value of .
1 mark, marked against the mark scheme
Find the value of .
1 mark, marked against the mark scheme