Outliers and cleaning data

Outliers and cleaning data

Spec 2.4 · Edexcel A-Level Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • An outlier lies a long way from the rest of the data; the question states the rule.
  • IQR rule: outside Q1−1.5 IQRQ_1-1.5\,\text{IQR} and Q3+1.5 IQRQ_3+1.5\,\text{IQR}. SD rule: outside xˉ±ks\bar x\pm ks.
  • A value exactly on a limit is not an outlier.
  • Decide if an outlier is an error or genuine, and justify what you do.
  • Cleaning data means correcting or removing errors and handling missing data.

What is an outlier?

Use the rule the question states; a value must lie beyond a limit to be an outlier.

An outlier is a value a long way from the rest of the data. Two common rules: more than 1.5×IQR1.5\times\text{IQR} above Q3Q_3 or below Q1Q_1, or more than kk standard deviations from the mean (often k=3k=3).

On a box plot, outliers are plotted individually and the whiskers stop at the most extreme non-outliers.

−50510152025303540lower limit −2Q₁Q₃upper limit 30outlier
Illustrative: Q₁ = 10, Q₃ = 18, IQR = 8, so the upper limit is 18 + 1.5 × 8 = 30. The value 35 is beyond it, so it is an outlier.
  • IQR ruleQ1−1.5 IQRQ_1-1.5\,\text{IQR} and Q3+1.5 IQRQ_3+1.5\,\text{IQR}
  • SD rulexˉ±ks\bar x\pm ks

A value lies exactly on the upper limit Q3+1.5 IQRQ_3+1.5\,\text{IQR}. Is it an outlier?

Applying the IQR rule

Show the calculation of both limits, then compare each suspect value with them.

Calculate 1.5×IQR1.5\times\text{IQR}, subtract it from Q1Q_1 and add it to Q3Q_3. Check both ends.

For Q1=14Q_1=14, Q3=22Q_3=22 the limits are 22 and 3434. The value 5050 is an outlier and 33 is not.

051015202530354045505523450

Worked example

Q1=14Q_1=14 and Q3=22Q_3=22. Are 50 and 3 outliers by the 1.5×IQR1.5\times\text{IQR} rule?

Q1=10Q_1=10 and Q3=18Q_3=18. What is the upper limit for outliers using the 1.5×IQR1.5\times\text{IQR} rule?

The mean and standard deviation rule

Values more than kk standard deviations from the mean are outliers.

For mean 805805 g and standard deviation 66 g, 3 standard deviations is 1818 g, so the limits are 787787 and 823823. A mass of 786786 is an outlier; 822822 is not.

An extreme value increases the mean and standard deviation, which widens the limits and can hide other unusual values, so it can be better to remove it and recalculate.

780785790795800805810815820825830787823786822
  • New meanold total−removed valuen−1\dfrac{\text{old total}-\text{removed value}}{n-1}

Worked example

Five values total 100. One of them, 50, is an outlier and is removed. Find the new mean.

A mean is 805805 g with standard deviation 66 g. Using 33 standard deviations, which is an outlier?

Dealing with outliers

Find the cause: correct or remove an error, but keep a genuine value.

A recording error, such as a misplaced decimal point, is corrected if the true value is known, otherwise removed. A genuine value is real data, so keep it, but consider the median and IQR.

Outliers strongly affect the mean, standard deviation and range; they have little effect on the median and IQR. Always justify what you do.

  1. 1

    Identify it

    Apply the stated rule.

  2. 2

    Find the cause

    Is it an error or a genuine value?

  3. 3

    Error

    Correct it if known, otherwise remove it.

  4. 4

    Genuine

    Keep it and consider the median and IQR.

  5. 5

    Justify

    State your decision and the reason.

Deciding what to do with an outlier

A genuine outlier is kept. Which pair of measures is least affected by it?

Cleaning data

Correct or remove errors and handle missing and duplicate values, recording what you changed.

Cleaning data removes or corrects values that would mislead the analysis. Record what you changed so others can see it.

For example 128 cm for a plant about 13 cm tall is implausible: correct it from the source or remove it. Replacing missing data with the mean adds values with no spread, so reduces the standard deviation.

  1. 1

    Look for errors

    correct implausible values from the source, otherwise remove them

  2. 2

    Handle missing data

    omit, estimate or re-collect

  3. 3

    Remove duplicates

    repeated records

  4. 4

    Record the changes

    so others can see what was done

Cleaning data before analysis.

Errors

Problem:
Impossible or implausible values
Action:
Correct from source, else remove

Missing data

Problem:
No value recorded
Action:
Omit, estimate or re-collect

Duplicates

Problem:
Repeated or inconsistent entries
Action:
Remove or standardise

Why does replacing missing values with the mean reduce the standard deviation?

Selecting and critiquing presentation

Pick the diagram that fits the question, then comment on location and spread in context.

A diagram can mislead if outliers are left in, axes are cut off or classes are badly chosen. Then interpret: comment on a measure of location and a measure of spread, in context.

  1. 1

    Pick the diagram

    box plot, histogram or cumulative frequency to fit the question

  2. 2

    Check it is fair

    outliers, cut-off axes and class choice

  3. 3

    Comment on location

    median or mean, in context

  4. 4

    Comment on spread

    IQR or standard deviation, in context

From choosing a diagram to a conclusion in context.

Box plot

Shows:
Median, quartiles and outliers
Best for:
Comparing groups

Histogram

Shows:
Shape: skew and peaks
Best for:
Continuous data

Cumulative frequency

Shows:
Medians and percentiles
Best for:
Grouped data

Which diagram is best for finding the median and percentiles of grouped data?

Try an exam question

The masses of bags of flour have mean 500500 g and standard deviation 44 g. An outlier is a value more than 33 standard deviations from the mean. Show that 485485 g is an outlier but 490490 g is not. The 485485 g bag is one of 1010 bags with total mass 50005000 g; find the mean mass of the other 99 bags.

[4 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

What is the upper limit above which a value is classed as an outlier?

1 mark, marked against the mark scheme

Revision notes