Outliers and data cleaning

Outliers and data cleaning

Spec L4 · AQA A-Level Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • An outlier is a value unusually far from the rest; the question tells you which rule to use.
  • Quartile rule: outside Q1−1.5×IQRQ_1-1.5\times\text{IQR} or Q3+1.5×IQRQ_3+1.5\times\text{IQR}. Standard deviation rule: more than kk standard deviations from the mean.
  • A value that meets a rule is only a possible outlier: check it before removing it.
  • Clean data: exclude missing values (never treat blanks as 0), fix or remove errors.
  • The median and IQR resist outliers; the mean and standard deviation do not.

What is an outlier?

An outlier is a value that lies unusually far from the rest of the data. A rule, usually given in the question, decides which values count.

Two common rules:

  • Quartile rule: more than 1.5×IQR1.5\times\text{IQR} above Q3Q_3 or below Q1Q_1.
  • Standard deviation rule: more than kk standard deviations (often 2 or 3) from the mean.

A value that meets the rule is only a possible outlier. It might be an error, or it might be genuine.

05101520253035404550121518upper fence 3542 (outlier)

Quartile rule

  • Above Q3+1.5×IQRQ_3+1.5\times\text{IQR}
  • Below Q1−1.5×IQRQ_1-1.5\times\text{IQR}

Standard deviation rule

  • Above xˉ+kσ\bar x+k\sigma
  • Below xˉ−kσ\bar x-k\sigma

Using the quartile rule with Q1=10Q_1=10 and Q3=18Q_3=18, is a value of 30 an outlier?

Testing for outliers

Calculate both boundaries, compare each value with them, then state your conclusion.

The quartile rule gives boundaries Q1−1.5×IQRQ_1-1.5\times\text{IQR} and Q3+1.5×IQRQ_3+1.5\times\text{IQR}. The standard deviation rule gives xˉ±kσ\bar x\pm k\sigma.

The number line shows the boundaries from the first example: values in the shaded ends are outliers.

010203040506070809.537.5

Worked example

Q1=20Q_1=20 and Q3=27Q_3=27. Ages of 71 and 17 are recorded. Which are outliers by the quartile rule?

Worked example

Mean 50, standard deviation 6. Using the 2-standard-deviation rule, are readings of 70 and 38.5 outliers?

Mean 40, standard deviation 5. Is 52 an outlier by the 2-standard-deviation rule?

Outliers in diagrams

Each diagram shows outliers differently. Name the outlier and suggest a cause in context.

Say what the outlier is and suggest a possible cause in context: a measurement error, a different group, or a rare but genuine event.

0246810120-910-1920-2930-3940-4950-5960-6970-79ClassFrequency
Histogram (illustrative data): the isolated bar far from the rest is a possible outlier.

Box plot

  • Marked individually, usually with a cross, beyond the whiskers

Scatter diagram

  • A point far from the pattern of the others

Histogram

  • An isolated bar far from the rest, or a very long tail

Stem-and-leaf

  • A value separated from the others by a gap

A scatter diagram of height against weight has one point far from the rest. What should you say?

Cleaning data

Deal with missing values, errors and outliers before analysis, and say what you did and why.

Data cleaning means dealing with problems before analysis.

Blanks are excluded and counted, never treated as 0. Impossible values (a negative spend, an age of 200) or codes like −1-1 are corrected if the true value can be found, otherwise removed.

Outliers are checked against the source: errors are corrected or removed; genuine ones are kept, and you can quote results with and without them.

  1. 1

    Missing values

    Exclude and count them; never treat as 0

  2. 2

    Errors

    Correct impossible values if the true value is known, otherwise remove

  3. 3

    Outliers

    Check against the source; keep genuine ones

  4. 4

    Report

    Say what you did and why

A cleaning routine to follow and report.

Missing data

  • Exclude from calculations
  • Say how many
  • Never treat as 0

Errors

  • Impossible values or codes
  • Correct if the true value is known
  • Otherwise remove

Outliers

  • Check against the source
  • Remove only if wrong
  • Keep genuine ones and comment

Worked example

A total of 25 777 over 194 entries includes three entries of −1-1 and one of 4500 (really 450). Find the cleaned mean.

A blank cell appears in a column of ages. What should you do?

Choosing and critiquing

Match the diagram to the data, and watch for outliers: the median and IQR resist them, the mean and standard deviation do not.

Histograms suit continuous grouped data, bar charts categories, scatter diagrams two variables and box plots comparing groups.

When critiquing, look for unequal class widths shown with frequency instead of frequency density, misleading or truncated axes, missing labels or units, and outliers hidden, exaggerated or uncleaned.

With an outlier present, the median is usually the better measure of a typical value.

024681012MeanMedianAverageValue
  • Without outlier
  • With 40 added
Data 4, 5, 6, 7, 8 compared with the same data plus 40: the mean jumps but the median hardly moves.

Resist outliers

  • Median
  • Interquartile range

Affected by outliers

  • Mean pulled towards them
  • Standard deviation inflated
  • Range distorted

Why is the median often better than the mean when there is a large outlier?

Try an exam question

A data set has Q1=24Q_1=24 and Q3=36Q_3=36. A value of 5858 is recorded. Using the quartile rule, decide whether 5858 is an outlier, and say what you should do next.

[4 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

Below which value would a daily number of visitors be an outlier?

1 mark, marked against the mark scheme

Revision notes