Outliers and cleaning data
Studying this course? Keep your progress in one place.
Key facts
- An outlier lies a long way from the rest of the data; the question states the rule.
- IQR rule: outside and . SD rule: outside .
- A value exactly on a limit is not an outlier.
- Decide if an outlier is an error or genuine, and justify what you do.
- Cleaning data means correcting or removing errors and handling missing data.
What is an outlier?
Use the rule the question states; a value must lie beyond a limit to be an outlier.
An outlier is a value a long way from the rest of the data. Two common rules: more than above or below , or more than standard deviations from the mean (often ).
On a box plot, outliers are plotted individually and the whiskers stop at the most extreme non-outliers.
- IQR rule and
- SD rule
A value lies exactly on the upper limit . Is it an outlier?
Applying the IQR rule
Show the calculation of both limits, then compare each suspect value with them.
Calculate , subtract it from and add it to . Check both ends.
For , the limits are and . The value is an outlier and is not.
Worked example
and . Are 50 and 3 outliers by the rule?
- 1
, so .
- 2
Lower limit ; upper limit .
- 3
, so 50 is an outlier.
- 4
, so 3 is not.
and . What is the upper limit for outliers using the rule?
The mean and standard deviation rule
Values more than standard deviations from the mean are outliers.
For mean g and standard deviation g, 3 standard deviations is g, so the limits are and . A mass of is an outlier; is not.
An extreme value increases the mean and standard deviation, which widens the limits and can hide other unusual values, so it can be better to remove it and recalculate.
- New mean
Worked example
Five values total 100. One of them, 50, is an outlier and is removed. Find the new mean.
- 1
New total from values.
- 2
New mean .
A mean is g with standard deviation g. Using standard deviations, which is an outlier?
Dealing with outliers
Find the cause: correct or remove an error, but keep a genuine value.
A recording error, such as a misplaced decimal point, is corrected if the true value is known, otherwise removed. A genuine value is real data, so keep it, but consider the median and IQR.
Outliers strongly affect the mean, standard deviation and range; they have little effect on the median and IQR. Always justify what you do.
- 1
Identify it
Apply the stated rule.
- 2
Find the cause
Is it an error or a genuine value?
- 3
Error
Correct it if known, otherwise remove it.
- 4
Genuine
Keep it and consider the median and IQR.
- 5
Justify
State your decision and the reason.
A genuine outlier is kept. Which pair of measures is least affected by it?
Cleaning data
Correct or remove errors and handle missing and duplicate values, recording what you changed.
Cleaning data removes or corrects values that would mislead the analysis. Record what you changed so others can see it.
For example 128 cm for a plant about 13 cm tall is implausible: correct it from the source or remove it. Replacing missing data with the mean adds values with no spread, so reduces the standard deviation.
- 1
Look for errors
correct implausible values from the source, otherwise remove them
- 2
Handle missing data
omit, estimate or re-collect
- 3
Remove duplicates
repeated records
- 4
Record the changes
so others can see what was done
| Errors | Missing data | Duplicates | |
|---|---|---|---|
| Problem | Impossible or implausible values | No value recorded | Repeated or inconsistent entries |
| Action | Correct from source, else remove | Omit, estimate or re-collect | Remove or standardise |
Errors
- Problem:
- Impossible or implausible values
- Action:
- Correct from source, else remove
Missing data
- Problem:
- No value recorded
- Action:
- Omit, estimate or re-collect
Duplicates
- Problem:
- Repeated or inconsistent entries
- Action:
- Remove or standardise
Why does replacing missing values with the mean reduce the standard deviation?
Selecting and critiquing presentation
Pick the diagram that fits the question, then comment on location and spread in context.
A diagram can mislead if outliers are left in, axes are cut off or classes are badly chosen. Then interpret: comment on a measure of location and a measure of spread, in context.
- 1
Pick the diagram
box plot, histogram or cumulative frequency to fit the question
- 2
Check it is fair
outliers, cut-off axes and class choice
- 3
Comment on location
median or mean, in context
- 4
Comment on spread
IQR or standard deviation, in context
| Box plot | Histogram | Cumulative frequency | |
|---|---|---|---|
| Shows | Median, quartiles and outliers | Shape: skew and peaks | Medians and percentiles |
| Best for | Comparing groups | Continuous data | Grouped data |
Box plot
- Shows:
- Median, quartiles and outliers
- Best for:
- Comparing groups
Histogram
- Shows:
- Shape: skew and peaks
- Best for:
- Continuous data
Cumulative frequency
- Shows:
- Medians and percentiles
- Best for:
- Grouped data
Which diagram is best for finding the median and percentiles of grouped data?
Try an exam question
The masses of bags of flour have mean g and standard deviation g. An outlier is a value more than standard deviations from the mean. Show that g is an outlier but g is not. The g bag is one of bags with total mass g; find the mean mass of the other bags.
[4 marks]
- [1], so the limits are and .
- [1], so is an outlier.
- [1] is within the limits, so it is not an outlier.
- [1]New mean g.
That's the notes covered.
Carry on to the next subtopic.
Try a question
What is the upper limit above which a value is classed as an outlier?
1 mark, marked against the mark scheme
What is the upper limit above which a value is classed as an outlier?
1 mark, marked against the mark scheme