Outliers and cleaning dataEdexcel A-Level Maths: Revision notes

An outlier is a value that lies a long way from the rest of the data. There is no single definition, so the question will state a rule. Two common rules:

• more than $1.5\times\text{IQR}$ above $Q_3$ or below $Q_1$, so the limits are $Q_1-1.5\,\text{IQR}$ and $Q_3+1.5\,\text{IQR}$;
• more than $k$ standard deviations from the mean (often $k=3$), so the limits are $\bar x\pm k s$. A value exactly on a limit is not outside it. On a box plot, outliers are plotted individually and the whiskers stop at the most extreme values that are not outliers.

Worked example: $Q_1=14$, $Q_3=22$. $\text{IQR}=8$, so $1.5\times8=12$. Lower limit $=14-12=2$; upper limit $=22+12=34$. A value of 50 is above 34, so it is an outlier. A value of 3 is above 2, so it is not. Always show the calculation of both limits, then compare each suspect value with them.

For mean $805$ g and standard deviation $6$ g, 3 standard deviations is $18$ g, so the limits are $805-18=787$ and $805+18=823$. A mass of $786$ is an outlier; $822$ is not. The mean and standard deviation are themselves affected by the outlier: an extreme value increases both, which widens the limits and can hide other unusual values. It can therefore be better to remove the outlier and recalculate. To find a mean after removing a value: new mean $=\frac{\text{old total}-\text{removed value}}{n-1}$.

First find the cause.

• A recording or measurement error (for example a misplaced decimal point): correct it if the true value is known, otherwise remove it.
• A genuine but unusual value: keep it, as it is real data, but consider reporting measures not affected by it, such as the median and IQR. Outliers strongly affect the mean, standard deviation and range; they have little effect on the median and IQR. Always justify what you do with an outlier.

Cleaning data means removing or correcting values that would mislead the analysis.

• Errors: values that are impossible or implausible (such as 128 cm for a plant that is about 13 cm tall); correct from the source if possible, otherwise remove.
• Missing data: either omit the case (this reduces the sample size) or replace it with an estimate such as the mean of the others (which is not a real measurement and reduces spread), or re-collect it.
• Duplicates or inconsistent entries: remove or standardise. Record what you changed so that others can see it.

Choose a diagram for the question:

• Box plot: shows median, quartiles and outliers, good for comparing groups.
• Histogram: shows shape (skew, peaks) of continuous data.
• Cumulative frequency: for medians and percentiles. Then interpret: comment on a measure of location and a measure of spread, in context. A diagram can mislead if outliers are left in, axes are cut off or classes are badly chosen.