Least squares regression and residuals

Least squares regression and residuals

Spec 1.1–2 · Edexcel A-Level Further Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • The least squares regression line y=a+bxy=a+bx minimises the sum of the squares of the residuals.
  • b=SxySxxb=\frac{S_{xy}}{S_{xx}} and a=yˉ−bxˉa=\bar y-b\bar x; the line passes through (xˉ,yˉ)(\bar x,\bar y).
  • Residual == observed −- predicted; residuals sum to zero.
  • RSS=Syy−Sxy2Sxx\text{RSS}=S_{yy}-\frac{S_{xy}^2}{S_{xx}}: smaller means a better fit.
  • Interpolation is reliable; extrapolation is not.

The least squares line

It is the line y=a+bxy=a+bx that minimises the sum of the squared vertical distances from the points.

The least squares regression line of yy on xx is the line y=a+bxy=a+bx that minimises the sum of the squares of the residuals, the vertical distances from the points to the line.

yy is the response variable and xx the explanatory variable, and the line should be used to predict yy from xx, not the other way round. To predict xx from yy you need a different line.

The derivation is not required; you must use the formulae for SxxS_{xx}, SyyS_{yy} and SxyS_{xy}.

12345612345678xymean point (3, 4)y = 1.3 + 0.9x
The least squares line for data (1,2), (2,3), (3,5), (4,4), (5,6) y = 1.3 + 0.9x (Sxy = 9, Sxx = 10, mean point (3, 4)).
  • SxxS_{xx}∑x2−(∑x)2n\sum x^2-\dfrac{(\sum x)^2}{n}
  • SyyS_{yy}∑y2−(∑y)2n\sum y^2-\dfrac{(\sum y)^2}{n}
  • SxyS_{xy}∑xy−∑x∑yn\sum xy-\dfrac{\sum x\sum y}{n}

You have the regression line of yy on xx. You want to predict xx from a given yy. What should you do?

Calculating the coefficients

Find b=SxySxxb=\frac{S_{xy}}{S_{xx}}, then a=yˉ−bxˉa=\bar y-b\bar x.

The gradient is b=SxySxxb=\frac{S_{xy}}{S_{xx}} and the intercept is a=yˉ−bxˉa=\bar y-b\bar x. The line always passes through (xˉ,yˉ)(\bar x,\bar y).

The gradient bb is the average change in yy for each unit increase in xx. The intercept aa is the predicted yy when x=0x=0, which may not be meaningful.

Compute SxxS_{xx} and SxyS_{xy} separately first and write them down; they also feed the RSS.

51015202530−2020406080100xy(19, 57)(0, −11.4)y = −11.4 + 3.6x
The line y = −11.4 + 3.6x passes through (x̄, ȳ) = (19, 57).
  • Gradientb=SxySxxb=\dfrac{S_{xy}}{S_{xx}}
  • Intercepta=yˉ−bxˉa=\bar y-b\bar x

Worked example

n=6n=6, ∑x=114\sum x=114, ∑y=342\sum y=342, ∑x2=2236\sum x^2=2236, ∑xy=6750\sum xy=6750. Find the regression line of yy on xx.

Sxy=30S_{xy}=30, Sxx=12S_{xx}=12, xˉ=4\bar x=4, yˉ=15\bar y=15. What is aa?

Residuals

A residual is observed minus predicted.

A residual is the difference between an observed value and the predicted value from the line: residual=yi−y^i=yi−(a+bxi)\text{residual}=y_i-\hat y_i=y_i-(a+bx_i).

A positive residual means the point is above the line; a negative residual means it is below.

Residuals from a least squares line always sum to zero. Subtracting the other way round (predicted minus observed) is a common mistake.

y = 1.3 + 0.9x+1.0−0.9PQ
Residual = observed − predicted: the point (3, 5) is 1.0 above the line (+1.0); (4, 4) is 0.9 below it (−0.9).
  • Residualyi−(a+bxi)y_i-(a+bx_i)

Worked example

For y=3+2.2xy=3+2.2x, find the residual for the observation (6,15)(6,15).

An observed value is 20 and the line predicts 17. What is the residual?

Residual sum of squares

The RSS is the quantity least squares minimises; a small RSS means a good fit.

The residual sum of squares is RSS=∑(yi−y^i)2=Syy−(Sxy)2Sxx\text{RSS}=\sum(y_i-\hat y_i)^2=S_{yy}-\frac{(S_{xy})^2}{S_{xx}}.

A small RSS relative to SyyS_{yy} means the line fits well.

Because it is a sum of squares, RSS is never negative; the second term of the formula can never exceed SyyS_{yy}.

00.20.40.60.810.040.010.04(1, 2)(2, 3)(3, 5)(4, 4)(5, 6)Data point (x, y)Squared residual
Squared residuals for the five points: RSS = 0.04 + 0.01 + 1 + 0.81 + 0.04 = 1.9 (= 10 − 9²/10).
  • Residual sum of squaresSyy−(Sxy)2SxxS_{yy}-\dfrac{(S_{xy})^2}{S_{xx}}

Worked example

Syy=920S_{yy}=920, Sxy=252S_{xy}=252, Sxx=70S_{xx}=70. Find the RSS.

For the same SyyS_{yy}, which RSS suggests the better fit?

Checking the fit and outliers

Small residuals with no pattern suggest a linear fit; a pattern suggests curvature, and one large residual an outlier.

Use residuals to judge whether a linear model is reasonable.

  • Small residuals with no pattern (a mixture of positive and negative): a linear fit is sensible.
  • A run of positive, then negative, then positive residuals suggests curvature, so a different model may be needed.
  • One residual much larger than the rest marks a possible outlier.

To refine a model, investigate the outlier. If there is a reason (a misreading, a recording error), remove it and recalculate the line and RSS. Do not remove a point only because it spoils the fit. In a comment, name the residual and the change in RSS.

  1. 1

    Spot it

    one residual much larger than the rest

  2. 2

    Investigate

    look for a misreading or recording error

  3. 3

    Remove only if justified

    not just because it spoils the fit

  4. 4

    Recalculate

    the line and the RSS

Dealing with a suspected outlier.

A residual plot shows positive, then negative, then positive residuals in a clear pattern. What does this suggest?

Using and interpreting the line

Predictions inside the data range are usually reliable; outside it they may be wildly wrong.

Predicting yy for an xx inside the data range is interpolation and is usually reliable.

Predicting outside the range is extrapolation and may be wildly wrong, because the relationship may not stay linear.

Always interpret the gradient in context: "for each extra hour of revision the score increases by 4.34.3 marks, on average". Check that predictions make sense, for example a test score over the maximum is impossible.

−2246810−224681012xydata startsdata endsfirst data pointlast data pointx = 10: extrapolatedy = 1.3 + 0.9x
The line y = 1.3 + 0.9x fitted to data with x from 1 to 5: inside this range (between the lines) is interpolation; beyond it is extrapolation.

Interpolation

  • x inside the data range
  • Usually reliable

Extrapolation

  • x outside the data range
  • May be wildly wrong

Data on revision hours ranges from 1 to 10 hours. Which prediction is the least reliable?

Try an exam question

A set of n=5n=5 points has ∑x=15\sum x=15, ∑y=20\sum y=20, ∑x2=55\sum x^2=55 and ∑xy=66\sum xy=66. Find the equation of the regression line of yy on xx.

[5 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

Find the value of bb.

1 mark, marked against the mark scheme

Revision notes