Type I and Type II errors and powerEdexcel A-Level Further Maths: Revision notes

A hypothesis test can go wrong in two ways. A Type I error is rejecting $\mathrm{H}_0$ when it is true. A Type II error is failing to reject $\mathrm{H}_0$ when it is false (so $\mathrm{H}_1$ is actually true). $\mathrm{P}(\text{Type I error})=\mathrm{P}(\text{reject }\mathrm{H}_0\mid\mathrm{H}_0\text{ true}),\qquad\mathrm{P}(\text{Type II error})=\mathrm{P}(\text{do not reject }\mathrm{H}_0\mid\mathrm{H}_1\text{ true}).$ Always say what the errors mean in the context of the question: for a coin, a Type I error is deciding it is biased when it is fair.

The size of a test is the probability of a Type I error, which is the probability that the test statistic falls in the critical region when $\mathrm{H}_0$ is true. For a discrete distribution the actual size is usually below the nominal significance level because the critical region has to be made of whole values. Example: $X\sim\mathrm{B}(10,p)$, $\mathrm{H}_0:p=0.5$, critical region $X\geq8$. Size $=\mathrm{P}(X\geq8\mid p=0.5)=\frac{45+10+1}{1024}=0.0547$. For a normal mean, $\bar X\sim\mathrm{N}\left(\mu,\frac{\sigma^2}{n}\right)$: reject if $\bar x<499$ with $\mu=500$, $\sigma=2$, $n=16$ gives $\mathrm{P}(Z<-2)=0.0228$.

To find $\mathrm{P}(\text{Type II error})$ you need a specific value of the parameter from $\mathrm{H}_1$. Use the same distribution family but with that value, and find the probability of landing in the acceptance region (the values where $\mathrm{H}_0$ is not rejected). Example: critical region $X\geq8$ for $\mathrm{B}(10,p)$ and true $p=0.7$: $\mathrm{P}(\text{Type II})=\mathrm{P}(X\leq7\mid p=0.7)=0.617$. Different true values of $p$ give different Type II probabilities, which is why a single value for the error needs a stated alternative.

The power of a test is the probability of rejecting $\mathrm{H}_0$ when it is false: $\text{power}=\mathrm{P}(\text{reject }\mathrm{H}_0\mid\mathrm{H}_1\text{ true})=1-\mathrm{P}(\text{Type II error}).$ The power function gives the power as a function of the parameter: $\text{power}(p)=\mathrm{P}(X\in\text{critical region}\mid p)$. At the value in $\mathrm{H}_0$ it equals the size of the test, and it rises as the true parameter moves further from the $\mathrm{H}_0$ value. For $\mathrm{B}(10,p)$ with critical region $X\geq8$: power at $p=0.6$ is $0.167$ and at $p=0.7$ is $0.383$.

A good test has small size and large power. Widening the critical region (for example $X\geq4$ instead of $X\geq5$) increases the power but also the size; narrowing it does the opposite. You cannot reduce both errors just by moving the boundary. The usual way to improve both is to increase the sample size. When you compare tests, compute the size of each and the power of each at the same alternative value, then judge against what is wanted. The same ideas work for the binomial, Poisson and normal distributions from A level Mathematics and Further Statistics 1.