Outliers and data cleaning
Studying this course? Keep your progress in one place.
Key facts
- An outlier is a value unusually far from the rest; the question tells you which rule to use.
- Quartile rule: outside or . Standard deviation rule: more than standard deviations from the mean.
- A value that meets a rule is only a possible outlier: check it before removing it.
- Clean data: exclude missing values (never treat blanks as 0), fix or remove errors.
- The median and IQR resist outliers; the mean and standard deviation do not.
What is an outlier?
An outlier is a value that lies unusually far from the rest of the data. A rule, usually given in the question, decides which values count.
Two common rules:
- Quartile rule: more than above or below .
- Standard deviation rule: more than standard deviations (often 2 or 3) from the mean.
A value that meets the rule is only a possible outlier. It might be an error, or it might be genuine.
Quartile rule
- Above
- Below
Standard deviation rule
- Above
- Below
Using the quartile rule with and , is a value of 30 an outlier?
Testing for outliers
Calculate both boundaries, compare each value with them, then state your conclusion.
The quartile rule gives boundaries and . The standard deviation rule gives .
The number line shows the boundaries from the first example: values in the shaded ends are outliers.
Worked example
and . Ages of 71 and 17 are recorded. Which are outliers by the quartile rule?
- 1
.
- 2
Upper boundary: .
- 3
Lower boundary: .
- 4
Compare: 71 is above 37.5, and 17 lies between 9.5 and 37.5.
Worked example
Mean 50, standard deviation 6. Using the 2-standard-deviation rule, are readings of 70 and 38.5 outliers?
- 1
Boundaries: , so 38 and 62.
- 2
70 is above 62.
- 3
38.5 is between 38 and 62.
Mean 40, standard deviation 5. Is 52 an outlier by the 2-standard-deviation rule?
Outliers in diagrams
Each diagram shows outliers differently. Name the outlier and suggest a cause in context.
Say what the outlier is and suggest a possible cause in context: a measurement error, a different group, or a rare but genuine event.
Box plot
- Marked individually, usually with a cross, beyond the whiskers
Scatter diagram
- A point far from the pattern of the others
Histogram
- An isolated bar far from the rest, or a very long tail
Stem-and-leaf
- A value separated from the others by a gap
A scatter diagram of height against weight has one point far from the rest. What should you say?
Cleaning data
Deal with missing values, errors and outliers before analysis, and say what you did and why.
Data cleaning means dealing with problems before analysis.
Blanks are excluded and counted, never treated as 0. Impossible values (a negative spend, an age of 200) or codes like are corrected if the true value can be found, otherwise removed.
Outliers are checked against the source: errors are corrected or removed; genuine ones are kept, and you can quote results with and without them.
- 1
Missing values
Exclude and count them; never treat as 0
- 2
Errors
Correct impossible values if the true value is known, otherwise remove
- 3
Outliers
Check against the source; keep genuine ones
- 4
Report
Say what you did and why
Missing data
- Exclude from calculations
- Say how many
- Never treat as 0
Errors
- Impossible values or codes
- Correct if the true value is known
- Otherwise remove
Outliers
- Check against the source
- Remove only if wrong
- Keep genuine ones and comment
Worked example
A total of 25 777 over 194 entries includes three entries of and one of 4500 (really 450). Find the cleaned mean.
- 1
Remove the three entries: total , entries .
- 2
Correct 4500 to 450: total .
- 3
Mean .
A blank cell appears in a column of ages. What should you do?
Choosing and critiquing
Match the diagram to the data, and watch for outliers: the median and IQR resist them, the mean and standard deviation do not.
Histograms suit continuous grouped data, bar charts categories, scatter diagrams two variables and box plots comparing groups.
When critiquing, look for unequal class widths shown with frequency instead of frequency density, misleading or truncated axes, missing labels or units, and outliers hidden, exaggerated or uncleaned.
With an outlier present, the median is usually the better measure of a typical value.
- Without outlier
- With 40 added
Resist outliers
- Median
- Interquartile range
Affected by outliers
- Mean pulled towards them
- Standard deviation inflated
- Range distorted
Why is the median often better than the mean when there is a large outlier?
Try an exam question
A data set has and . A value of is recorded. Using the quartile rule, decide whether is an outlier, and say what you should do next.
[4 marks]
- [1].
- [1]Upper boundary .
- [1], so it is an outlier.
- [1]Check it against the source: correct or remove it if it is an error, keep it and comment if it is genuine.
That's the notes covered.
Carry on to the next subtopic.
Try a question
Below which value would a daily number of visitors be an outlier?
1 mark, marked against the mark scheme
Below which value would a daily number of visitors be an outlier?
1 mark, marked against the mark scheme