Type I and Type II errors

Type I and Type II errors

Spec SC1 · AQA A-Level Further Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • A Type I error is rejecting H0\mathrm{H}_0 when it is true; a Type II error is not rejecting it when it is false.
  • Describe each error in context: what is concluded and what is really true.
  • P(Type I)=P(critical region)\mathrm{P}(\text{Type I})=\mathrm{P}(\text{critical region}) calculated with the parameter from H0\mathrm{H}_0.
  • A larger critical region raises the probability of a Type I error.
  • At least one false rejection in nn tests: 1−(1−α)n1-(1-\alpha)^n.

Type I and Type II errors

A test can reject a true H0\mathrm{H}_0 (Type I) or fail to reject a false one (Type II).

A hypothesis test is a decision based on sample evidence, so it can be wrong.

  • A Type I error is rejecting H0\mathrm{H}_0 when it is true.
  • A Type II error is failing to reject H0\mathrm{H}_0 when it is false.

You must be able to define each in the context of the question and calculate the probability of a Type I error.

−4−3−2−11234560.20.40.60.81xycritical valueH₀ trueH₀ false
Illustrative: solid = statistic if H₀ true, dashed = if H₀ false. Shaded tail = Type I error probability.

H0 is actually true

Reject H0:
Type I error
Do not reject H0:
Correct decision

H0 is actually false

Reject H0:
Correct decision
Do not reject H0:
Type II error

A test rejects H0 although H0 is true. What type of error is this?

Describing errors in context

Say what the test would conclude and what is actually true, in the words of the situation.

Write what the test would conclude and what is actually true, using the words of the situation, not just "H0\mathrm{H}_0" and "H1\mathrm{H}_1".

A seed company claims p=0.8p=0.8 and a gardener tests H1:p<0.8\mathrm{H}_1:p<0.8.

  • Type I: concluding that the probability of germination is below 0.80.8 when it is in fact 0.80.8.
  • Type II: concluding there is no evidence the probability is below 0.80.8 when it is in fact lower.
0.40.450.50.550.60.650.70.750.80.850.90.951H₀: p = 0.8

Type I error

Test concludes:
Germination probability is below 0.8
Reality:
It is actually 0.8

Type II error

Test concludes:
No evidence it is below 0.8
Reality:
It is actually lower

A factory claims 2% of bolts are faulty (H0:p=0.02\mathrm{H}_0:p=0.02) and a customer tests H1:p>0.02\mathrm{H}_1:p>0.02. Describe a Type II error.

Probability of a Type I error

Find the probability of landing in the critical region when H0\mathrm{H}_0 is true, using the parameter from H0\mathrm{H}_0.

The probability of a Type I error is the probability that the test statistic falls in the critical region when H0\mathrm{H}_0 is true. It is also the actual significance level.

Always use the value of the parameter given by H0\mathrm{H}_0. The method is the same for binomial and Poisson tests.

For a two-tailed critical region, add both tails.

00.010.020.030.04P(X ≤ 5)P(X ≥ 15)Type I errorProbabilityValue
Illustrative two-tailed example, X ~ B(20, 0.5), critical region X ≤ 5 or X ≥ 15: add both tails.
  • P(Type I error)P(test statistic in critical region∣H0)\mathrm{P}(\text{test statistic in critical region}\mid\mathrm{H}_0)

Worked example

H0:p=0.3\mathrm{H}_0:p=0.3 against H1:p>0.3\mathrm{H}_1:p>0.3 with X∼B(20,p)X\sim\mathrm{B}(20,p) and critical region X≥10X\ge10. Find the probability of a Type I error.

Worked example

H0:λ=6\mathrm{H}_0:\lambda=6, H1:λ<6\mathrm{H}_1:\lambda<6, reject H0\mathrm{H}_0 if Y≤2Y\le2. Find the probability of a Type I error.

H0:λ=5\mathrm{H}_0:\lambda=5, critical region X≤1X\le1 for X∼Po(λ)X\sim\mathrm{Po}(\lambda). Which expression gives the probability of a Type I error?

Choosing a critical region

Take the largest region whose probability is at most the significance level.

To test at the 5% level, take the largest critical region whose probability is at most 0.050.05.

Making the critical region larger increases the probability of a Type I error.

The actual significance level can be well below the nominal significance level of 5% because the distribution is discrete.

  1. 1

    Try a boundary

    find its probability under H0

  2. 2

    Compare

    is it at most the significance level?

  3. 3

    Adjust

    if too large, move one value inwards

  4. 4

    State

    the region and its actual significance level

Finding a critical region

Worked example

X∼B(15,0.8)X\sim\mathrm{B}(15,0.8) and H1:p<0.8\mathrm{H}_1:p<0.8. Find the critical region at the 5% level and the actual significance level.

The critical region is made larger. What happens to the probability of a Type I error?

Repeated testing

Each test has probability α\alpha of a false rejection, so the chance of at least one is 1−(1−α)n1-(1-\alpha)^n.

If a true H0\mathrm{H}_0 is tested several times independently, each test has probability α\alpha of a false rejection.

The probability of at least one wrong rejection in nn tests is

1−(1−α)n.1-(1-\alpha)^n.

With α=0.0397\alpha=0.0397 and n=10n=10 this is 1−0.960310=0.3331-0.9603^{10}=0.333. Many tests make at least one false alarm quite likely.

00.10.20.30.40.51251020Number of tests, nProbability
Probability of at least one Type I error when alpha = 0.0397

Each test has α=0.05\alpha=0.05. What is the probability of at least one Type I error in 2 independent tests of a true H0\mathrm{H}_0?

Try an exam question

A test of H0:p=0.3\mathrm{H}_0:p=0.3 against H1:p>0.3\mathrm{H}_1:p>0.3 uses X∼B(20,p)X\sim\mathrm{B}(20,p) and rejects H0\mathrm{H}_0 if X≥10X\ge10. Find the probability of a Type I error and state what a Type I error means in this test.

[3 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

Which of the following describes a Type I error?

1 mark, marked against the mark scheme

Revision notes