Type I and Type II errors and power
Studying this course? Keep your progress in one place.
Key facts
- A Type I error rejects when it is true; a Type II error fails to reject when it is false.
- The size of a test is , found assuming is true.
- needs a specific value of the parameter from .
- Power ; the power function gives it for every true value.
- A good test has small size and large power; increasing the sample size improves both.
Type I and Type II errors
Rejecting a true is a Type I error; failing to reject a false is a Type II error.
A Type I error is rejecting when it is true. A Type II error is failing to reject when it is false (so is actually true).
and .
Always say what the errors mean in context: for a coin, a Type I error is deciding it is biased when it is fair. A Type II error is not "accepting wrongly".
| H₀ true | H₀ false | |
|---|---|---|
| Reject H₀ | Type I error | Correct decision |
| Do not reject H₀ | Correct decision | Type II error |
H₀ true
- Reject H₀:
- Type I error
- Do not reject H₀:
- Correct decision
H₀ false
- Reject H₀:
- Correct decision
- Do not reject H₀:
- Type II error
A coin is actually biased, but the test does not reject the claim that it is fair. Which error is this?
The size of a test
The size is the probability the test statistic lands in the critical region when is true.
The size of a test is the probability of a Type I error: the probability that the test statistic falls in the critical region when is true.
For a discrete distribution the actual size is usually below the nominal significance level because the critical region has to be made of whole values.
For a normal mean, . Reject if with , , : the standard deviation of is , so and the size is .
- Size
Worked example
, , critical region . Find the size of the test.
- 1
Assume is true, so .
- 2
Size .
A test rejects when . Which probability is its size?
Probability of a Type II error
Use a specific value from and find the probability of landing in the acceptance region.
To find you need a specific value of the parameter from .
Use the same distribution family but with that value, and find the probability of landing in the acceptance region, the values where is not rejected.
Different true values of give different Type II probabilities, so a single value needs a stated alternative. Using the value from is a common mistake.
- Type II error
Worked example
Critical region for . Find when the true .
- 1
The acceptance region is .
- 2
Use and find .
Which value of do you use to calculate a Type II error probability?
Power and the power function
Power is the probability of correctly rejecting : one minus the Type II error probability.
The power of a test is the probability of rejecting when it is false: , at the same true parameter value.
The power function gives the power as a function of the parameter: .
At the value in it equals the size of the test, and it rises as the true parameter moves further from the value.
For with critical region : power at is and at is .
- Power
The probability of a Type II error is at . What is the power at ?
Effectiveness and trade-offs
A good test has small size and large power; moving the boundary trades one against the other.
A good test has small size and large power.
Widening the critical region increases the power but also the size; narrowing it does the opposite. You cannot reduce both errors just by moving the boundary. The usual way to improve both is to increase the sample size.
To compare tests, compute the size of each and the power of each at the same alternative value, then judge against what is wanted. For "evaluate" questions, quote both errors for each test and give a recommendation. The same ideas work for binomial, Poisson and normal distributions.
- 1
Find the size
of each test, assuming the null hypothesis is true
- 2
Find the power
of each at the same alternative value
- 3
Compare
against what the question wants
- 4
Recommend
and justify using both errors
You widen the critical region of a test. What happens?
Try an exam question
. A test of against rejects if . Find the size of the test, and the probability of a Type II error when the true value of is .
[5 marks]
- [1]Under , .
- [1]Size .
- [1]The acceptance region is .
- [1]Uses .
- [1].
That's the notes covered.
Carry on to the next subtopic.
Try a question
Find the size of the test.
1 mark, marked against the mark scheme
Find the size of the test.
1 mark, marked against the mark scheme