Type I and Type II errors and power

Type I and Type II errors and power

Spec 8.1 · Edexcel A-Level Further Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • A Type I error rejects H0\mathrm{H}_0 when it is true; a Type II error fails to reject H0\mathrm{H}_0 when it is false.
  • The size of a test is P(Type I error)\mathrm{P}(\text{Type I error}), found assuming H0\mathrm{H}_0 is true.
  • P(Type II error)\mathrm{P}(\text{Type II error}) needs a specific value of the parameter from H1\mathrm{H}_1.
  • Power =1−P(Type II error)=1-\mathrm{P}(\text{Type II error}); the power function gives it for every true value.
  • A good test has small size and large power; increasing the sample size improves both.

Type I and Type II errors

Rejecting a true H0\mathrm{H}_0 is a Type I error; failing to reject a false H0\mathrm{H}_0 is a Type II error.

A Type I error is rejecting H0\mathrm{H}_0 when it is true. A Type II error is failing to reject H0\mathrm{H}_0 when it is false (so H1\mathrm{H}_1 is actually true).

P(Type I error)=P(reject H0∣H0 true)\mathrm{P}(\text{Type I error})=\mathrm{P}(\text{reject }\mathrm{H}_0\mid\mathrm{H}_0\text{ true}) and P(Type II error)=P(do not reject H0∣H1 true)\mathrm{P}(\text{Type II error})=\mathrm{P}(\text{do not reject }\mathrm{H}_0\mid\mathrm{H}_1\text{ true}).

Always say what the errors mean in context: for a coin, a Type I error is deciding it is biased when it is fair. A Type II error is not "accepting H1\mathrm{H}_1 wrongly".

−3−2−11234560.10.20.30.4xycritical valueH₀ trueH₁ true
Illustrative: rejecting H₀ when it is true (shaded tail) is a Type I error. A Type II error happens when H₁ is true but x lands left of the line.

H₀ true

Reject H₀:
Type I error
Do not reject H₀:
Correct decision

H₀ false

Reject H₀:
Correct decision
Do not reject H₀:
Type II error

A coin is actually biased, but the test does not reject the claim that it is fair. Which error is this?

The size of a test

The size is the probability the test statistic lands in the critical region when H0\mathrm{H}_0 is true.

The size of a test is the probability of a Type I error: the probability that the test statistic falls in the critical region when H0\mathrm{H}_0 is true.

For a discrete distribution the actual size is usually below the nominal significance level because the critical region has to be made of whole values.

For a normal mean, Xˉ∼N(μ,σ2n)\bar X\sim\mathrm{N}\left(\mu,\frac{\sigma^2}{n}\right). Reject if xˉ<499\bar x<499 with μ=500\mu=500, σ=2\sigma=2, n=16n=16: the standard deviation of Xˉ\bar X is 0.50.5, so z=−2z=-2 and the size is P(Z<−2)=0.0228\mathrm{P}(Z<-2)=0.0228.

498498.5499499.5500500.5501501.55020.10.20.30.40.50.60.70.80.9xyx̄ = 499X̄ under H₀
X̄ ~ N(500, 0.5²) under H₀. The shaded tail below 499 (z = −2) has area 0.0228, the size of the test.
  • SizeP(reject H0∣H0 true)\mathrm{P}(\text{reject }\mathrm{H}_0\mid\mathrm{H}_0\text{ true})

Worked example

X∼B(10,p)X\sim\mathrm{B}(10,p), H0:p=0.5\mathrm{H}_0:p=0.5, critical region X≥8X\geq8. Find the size of the test.

A test rejects H0\mathrm{H}_0 when X≥8X\geq8. Which probability is its size?

Probability of a Type II error

Use a specific value from H1\mathrm{H}_1 and find the probability of landing in the acceptance region.

To find P(Type II error)\mathrm{P}(\text{Type II error}) you need a specific value of the parameter from H1\mathrm{H}_1.

Use the same distribution family but with that value, and find the probability of landing in the acceptance region, the values where H0\mathrm{H}_0 is not rejected.

Different true values of pp give different Type II probabilities, so a single value needs a stated alternative. Using the value from H0\mathrm{H}_0 is a common mistake.

−3−2−11234560.10.20.30.4xycritical valueH₀ trueH₁ true
Illustrative: with a value from H₁, a Type II error is the area of the H₁ curve left of the critical value (shaded).
  • Type II errorP(acceptance region∣a value from H1)\mathrm{P}(\text{acceptance region}\mid\text{a value from }\mathrm{H}_1)

Worked example

Critical region X≥8X\geq8 for B(10,p)\mathrm{B}(10,p). Find P(Type II error)\mathrm{P}(\text{Type II error}) when the true p=0.7p=0.7.

Which value of pp do you use to calculate a Type II error probability?

Power and the power function

Power is the probability of correctly rejecting H0\mathrm{H}_0: one minus the Type II error probability.

The power of a test is the probability of rejecting H0\mathrm{H}_0 when it is false: power=1−P(Type II error)\text{power}=1-\mathrm{P}(\text{Type II error}), at the same true parameter value.

The power function gives the power as a function of the parameter: power(p)=P(X∈critical region∣p)\text{power}(p)=\mathrm{P}(X\in\text{critical region}\mid p).

At the value in H0\mathrm{H}_0 it equals the size of the test, and it rises as the true parameter moves further from the H0\mathrm{H}_0 value.

For B(10,p)\mathrm{B}(10,p) with critical region X≥8X\geq8: power at p=0.6p=0.6 is 0.1670.167 and at p=0.7p=0.7 is 0.3830.383.

00.20.40.60.810.50.60.70.80.9True value of pPower
Power function for B(10, p) with critical region X ≥ 8.
  • Power1−P(Type II error)1-\mathrm{P}(\text{Type II error})

The probability of a Type II error is 0.6170.617 at p=0.7p=0.7. What is the power at p=0.7p=0.7?

Effectiveness and trade-offs

A good test has small size and large power; moving the boundary trades one against the other.

A good test has small size and large power.

Widening the critical region increases the power but also the size; narrowing it does the opposite. You cannot reduce both errors just by moving the boundary. The usual way to improve both is to increase the sample size.

To compare tests, compute the size of each and the power of each at the same alternative value, then judge against what is wanted. For "evaluate" questions, quote both errors for each test and give a recommendation. The same ideas work for binomial, Poisson and normal distributions.

  1. 1

    Find the size

    of each test, assuming the null hypothesis is true

  2. 2

    Find the power

    of each at the same alternative value

  3. 3

    Compare

    against what the question wants

  4. 4

    Recommend

    and justify using both errors

Comparing two tests.

You widen the critical region of a test. What happens?

Try an exam question

X∼B(10,p)X\sim\mathrm{B}(10,p). A test of H0:p=0.5\mathrm{H}_0:p=0.5 against H1:p>0.5\mathrm{H}_1:p>0.5 rejects H0\mathrm{H}_0 if X≥8X\geq8. Find the size of the test, and the probability of a Type II error when the true value of pp is 0.70.7.

[5 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

Find the size of the test.

1 mark, marked against the mark scheme

Revision notes