Outliers and data cleaningAQA A-Level Maths: Revision notes

An outlier is a value that lies unusually far from the rest of the data. There is no single universal rule, so a question will normally tell you which rule to use. Two common ones:

• Quartile rule: a value is an outlier if it is more than $1.5\times\text{IQR}$ above $Q_3$ or more than $1.5\times\text{IQR}$ below $Q_1$.
• Standard deviation rule: a value is an outlier if it is more than $k$ standard deviations (often 2 or 3) from the mean. A value that meets the rule is only a possible outlier. It might be an error, or it might be genuine.

Quartile rule worked example: $Q_1=20$, $Q_3=27$. IQR $=7$. Upper boundary $27+1.5\times7=37.5$; lower boundary $20-10.5=9.5$. An age of 71 is above 37.5, so it is an outlier, while 17 is within the boundaries. Standard deviation rule: mean 50, standard deviation 6, 2-standard-deviation rule gives boundaries $50\pm12=38$ and $62$. A reading of 70 is an outlier; 38.5 is not. Always calculate both boundaries and compare, then state your conclusion.

• Box plot: outliers are marked individually (usually with a cross) beyond the whiskers.
• Scatter diagram: an outlier is a point far from the general pattern of the others.
• Histogram: an isolated bar a long way from the rest, or a very long tail.
• Stem-and-leaf: a value separated from the others by a gap. When you interpret a diagram, say what the outlier is and suggest a possible cause in context, such as a measurement error, a different group, or a rare but genuine event.

Data cleaning means dealing with problems before analysis.

• Missing data (blank cells): exclude them from calculations and say how many; do not treat blanks as 0.
• Errors: impossible values (such as a negative spend or an age of 200) or codes like $-1$. Correct them if the true value can be found (for example from the source or an obvious typing slip), otherwise remove them.
• Outliers: check them against the source. If they are errors, correct or remove them. If they are genuine, keep them and comment on their effect, for example by quoting results with and without the outlier. Example: a total of 25 777 over 194 entries includes three entries of $-1$ and one of 4500 (really 450). The cleaned total is $25777+3-4500+450=21730$ over 191 entries, mean $113.77$.

Match the diagram to the data: histograms for continuous grouped data, bar charts for categories, scatter diagrams for two variables, box plots to compare groups. When critiquing, look for:

• Unequal class widths shown with frequency instead of frequency density.
• Misleading or truncated axes, or missing labels and units.
• Outliers hidden or exaggerated, or data that has not been cleaned. Measures of average are affected by outliers: the mean is pulled towards them, while the median and IQR resist them. With an outlier present, the median is usually the better measure of a typical value.