Type II errors and power

Type II errors and power

Spec SC2 · AQA A-Level Further Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • A Type II error is not rejecting H0\mathrm{H}_0 when it is false; its probability needs a stated true value.
  • Use the distribution for the true parameter value, not the H0\mathrm{H}_0 value.
  • Power =1−P(Type II error)=1-\mathrm{P}(\text{Type II error}).
  • Power rises when the true value is further from H0\mathrm{H}_0, or the sample is larger.
  • Enlarging the critical region lowers P\mathrm{P}(Type II) but raises P\mathrm{P}(Type I).

Type II errors and their probability

Find the probability that the test statistic is outside the critical region, using the true parameter value.

A Type II error is failing to reject H0\mathrm{H}_0 when H0\mathrm{H}_0 is false.

To find its probability you need a specific true value of the parameter:

P(Type II)=P(test statistic outside the critical region∣true value)\mathrm{P}(\text{Type II})=\mathrm{P}(\text{test statistic outside the critical region}\mid\text{true value})

Use the distribution for the true value, not the value in H0\mathrm{H}_0. For a Type I error it is the other way round.

−3−2−11234560.20.40.60.81xycritical valueif H0 is truetrue value
Illustrative: the shaded part of the true-value curve lies outside the critical region. Its area is P(Type II error).

H0 is true

Reject H0:
Type I error
Do not reject H0:
Correct

H0 is false

Reject H0:
Correct (the power)
Do not reject H0:
Type II error

Which distribution do you use to find the probability of a Type II error?

The power of a test

The probability of correctly rejecting a false H0\mathrm{H}_0, for a given true value.

The power of a test is the probability of rejecting H0\mathrm{H}_0 when it is false:

power=1−P(Type II error)=P(statistic in the critical region∣true value)\text{power}=1-\mathrm{P}(\text{Type II error})=\mathrm{P}(\text{statistic in the critical region}\mid\text{true value})

Power depends on the true value, so a test has a different power for each possible alternative. High power means the test is good at detecting a real change.

00.20.40.60.810.50.60.70.80.9True value of pPower
Power of the test H0: p = 0.5, H1: p > 0.5, with X ~ B(10, p), rejecting H0 if X ≥ 8. The further the true p is from 0.5, the higher the power.
  • Power1−P(Type II)1-\mathrm{P}(\text{Type II})
  • Power depends onthe true value

For a given true value, P(Type II error)=0.35\mathrm{P}(\text{Type II error})=0.35. What is the power?

Binomial and Poisson tests

Work out the non-rejection region, then use the distribution with the true parameter.

Binomial. H0:p=0.5\mathrm{H}_0:p=0.5, H1:p>0.5\mathrm{H}_1:p>0.5, X∼B(10,p)X\sim\mathrm{B}(10,p), reject if X≥8X\ge8. True p=0.7p=0.7: P(Type II)=P(X≤7)=0.6172\mathrm{P}(\text{Type II})=\mathrm{P}(X\le7)=0.6172.

Poisson. H0:λ=5\mathrm{H}_0:\lambda=5, reject if Y≥10Y\ge10. True λ=8\lambda=8: P(Type II)=P(Y≤9)=0.7166\mathrm{P}(\text{Type II})=\mathrm{P}(Y\le9)=0.7166.

Write the distribution with the true parameter first. The non-rejection region is the complement of the critical region: take care with the boundary.

00.050.10.150.20.250.0010.00980.00980.0010.00010.00140.009012345678910XP(X = x)
  • p = 0.5
  • p = 0.7
B(10, p) probabilities. The test rejects H0 if X ≥ 8; with the true p = 0.7 the Type II error is the sum of the p = 0.7 bars for X ≤ 7 (0.6172).

Binomial

Test:
H0: p = 0.5, reject X ≥ 8
True value:
p = 0.7
P(Type II):
P(X ≤ 7) = 0.6172
Power:
0.3828

Poisson

Test:
H0: λ = 5, reject Y ≥ 10
True value:
λ = 8
P(Type II):
P(Y ≤ 9) = 0.7166
Power:
0.2834

Worked example

H0:p=0.3\mathrm{H}_0:p=0.3, H1:p>0.3\mathrm{H}_1:p>0.3, X∼B(20,p)X\sim\mathrm{B}(20,p). The test rejects H0\mathrm{H}_0 if X≥10X\ge10. The true value is p=0.5p=0.5. Find P\mathrm{P}(Type II error) and the power.

A test rejects H0\mathrm{H}_0 if X≥8X\ge8. Which expression gives P\mathrm{P}(Type II error)?

Normal tests

Use xˉ∼N(μ,σ2n)\bar x\sim\mathrm{N}\left(\mu,\frac{\sigma^2}{n}\right) with μ\mu the true mean, and find the probability of falling in the critical region.

For a test of a mean with σ\sigma known, xˉ∼N(μ,σ2n)\bar x\sim\mathrm{N}\left(\mu,\frac{\sigma^2}{n}\right) with μ\mu the true mean.

One-tailed: H0:μ=500\mathrm{H}_0:\mu=500, H1:μ<500\mathrm{H}_1:\mu<500, σ=4\sigma=4, n=16n=16, reject if xˉ<498\bar x<498. True mean 497497: xˉ∼N(497,12)\bar x\sim\mathrm{N}(497,1^2), so power =P(Z<1)=0.841=\mathrm{P}(Z<1)=0.841.

Two-tailed: reject if xˉ<993\bar x<993 or xˉ>1007\bar x>1007, true mean 10101010, sd 44: P(Type II)=P(993≤xˉ≤1007)=0.227\mathrm{P}(\text{Type II})=\mathrm{P}(993\le\bar x\le1007)=0.227. This is the non-rejection region.

4944954964974984995005015025030.10.20.30.4xycritical value 498under H₀: mean 500true mean 497
H₀ (dashed) and the true distribution; the area left of 498 is the power

Worked example

H0:μ=50\mathrm{H}_0:\mu=50, H1:μ>50\mathrm{H}_1:\mu>50, standard deviation of xˉ\bar x equal to 22. Reject if xˉ>54\bar x>54. The true mean is 5656. Find the power.

In the example above, which region gives the Type II error?

What affects power

Power is higher for a true value further from H0\mathrm{H}_0, a larger sample or a larger critical region; the last also raises P\mathrm{P}(Type I).

Power is higher when:

  • the true value is further from the H0\mathrm{H}_0 value;
  • the sample size is larger, so xˉ\bar x has a smaller standard deviation;
  • the critical region is larger, but this also raises the probability of a Type I error.

So there is a trade-off: enlarging the critical region lowers P\mathrm{P}(Type II) but raises P\mathrm{P}(Type I). Only a bigger sample can improve both.

00.20.40.60.810.50.60.70.80.9True value of pPower
Power of 'reject H₀ if X ≥ 8', X ~ B(10, p), for different true p

What happens to the power of a test if the true value moves further from the H₀ value?

Try an exam question

A test of H0:p=0.3\mathrm{H}_0:p=0.3 against H1:p>0.3\mathrm{H}_1:p>0.3 uses X∼B(20,p)X\sim\mathrm{B}(20,p) and rejects H0\mathrm{H}_0 if X≥10X\ge10. The true value of pp is 0.50.5. Find the probability of a Type II error and the power of the test.

[4 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

Find the probability of a Type II error when the true value is p=0.7p=0.7.

1 mark, marked against the mark scheme

Revision notes