4.12 Data collection, reliability and validity
Studying this course? Keep your progress in one place.
Key facts
- Choose relevant variables and a sample that represents the population, or the data will be biased.
- Questions should be neutral, precise and give consistent answer choices.
- For a test, expected frequencies must be above : combine categories if not.
- Goodness of fit has , where is the number of parameters estimated from the data.
- Reliable means consistent; valid means measures the right thing. Reliable does not mean valid.
Designing a valid method
Start from the question: collect relevant data from a sample that represents the population.
A survey or questionnaire collects data from a sample. Choose the relevant variables from the many that could be recorded: collecting everything makes analysis messy, collecting too little makes a conclusion impossible.
The sample must represent the population. Asking only people who are easy to reach (students leaving the canteen) can introduce bias.
Check the data type suits the analysis: numerical data for means and regression, categorical data for tests.
- 1
Question
What must the data answer?
- 2
Variables
Record only the relevant ones
- 3
Sample
Representative, not just convenient
- 4
Data type
Suits the analysis planned
A survey about school meals only asks students leaving the canteen. What is the main problem?
Writing good questions
Good questions are neutral, precise and offer balanced answer choices.
A biased question leads the respondent: "Don't you agree that the new menu is far better?". A neutral one asks "How satisfied are you with the new menu?".
Personal questions (age, income, health) may get untruthful answers, so ask only what is needed and consider ranges.
Structured questions give consistent choices, such as a five-point scale; they are easy to count. Unstructured questions allow free answers: rich but hard to analyse. Choices should cover every answer and not overlap.
Precise wording means exact units: "How many hours did you sleep last night?".
Biased
- Don't you agree the canteen is far too expensive?
- Surely the prices are unfair?
Unbiased
- How would you rate the canteen prices from very low to very high?
- How satisfied are you with the new menu?
| Structured | Unstructured | |
|---|---|---|
| Advantage | Easy to count and compare | Rich, detailed answers |
| Disadvantage | May not include the respondent's answer | Hard to analyse |
Structured
- Advantage:
- Easy to count and compare
- Disadvantage:
- May not include the respondent's answer
Unstructured
- Advantage:
- Rich, detailed answers
- Disadvantage:
- Hard to analyse
Which question is unbiased?
Categories for a chi-squared test
Group the values into justified categories, and combine categories until every expected frequency is above .
To use a test on numerical data (for example hours slept), sort the values into categories and count each. Choose categories that make sense and justify the choice, for example equal widths.
The test needs expected frequencies above in every category. If one is smaller, combine categories with a neighbour and recalculate.
The rule applies to the expected values, not the observed ones.
- Expected frequency
- Rule of 5every expected frequency
Worked example
In a table of students, the "under 5 hours" row total is and the "alert" column total is . Is the expected frequency large enough?
- 1
.
- 2
, so the rule is broken.
- 3
Combine "under 5" with "5 to under 6" and recalculate all expected values.
Which values must be greater than for the test?
Degrees of freedom with estimates
Subtract for the total and more for each parameter estimated from the data.
For a goodness of fit test with categories (after combining), , where is the number of parameters estimated from the data, such as for a binomial or the mean for a Poisson. If the parameters are given, .
For a contingency table, .
Use your GDC for and the -value, and reject if the significance level.
- Goodness of fit
- Contingency table
Worked example
A binomial is fitted with estimated from the data. After combining there are categories. Find .
- 1
categories after combining.
- 2
One parameter () was estimated, so .
- 3
.
A Poisson model with the mean estimated from the data is tested with categories. Find .
Reliability
A test is reliable if it gives consistent results when repeated under the same conditions.
Reliable says nothing about whether the test measures the right thing.
Test-retest: give the same test to the same people twice and find the correlation between the scores. A value of close to means high reliability.
Parallel forms: give two different but equivalent versions to the same people and correlate the scores.
- 1
First sitting
Give the test to a group
- 2
Second sitting
Same test, same people, same conditions
- 3
Correlate
Find r between the two sets of scores
- 4
Interpret
r close to 1 means high reliability
Test-retest
- Same test, same people, twice
- Correlate the two sets of scores
Parallel forms
- Two equivalent versions, same people
- Correlate the two sets of scores
A test-retest correlation is . What does this show?
Validity versus reliability
A test is valid if it measures what it is meant to measure; it can be reliable without being valid.
Content validity: the questions cover all the relevant content, usually judged by experts. Criterion-related validity: scores agree with an independent, accepted measure of the same thing.
A bathroom scale that is always kg too heavy is reliable (consistent) but not valid (accurate). A test that is not reliable cannot be valid.
| Reliable | Valid | |
|---|---|---|
| Question it answers | Are the results consistent? | Does it measure the right thing? |
| Example of failure | Scores change a lot between two sittings | A scale always 3 kg too heavy |
Reliable
- Question it answers:
- Are the results consistent?
- Example of failure:
- Scores change a lot between two sittings
Valid
- Question it answers:
- Does it measure the right thing?
- Example of failure:
- A scale always 3 kg too heavy
A scale always reads 3 kg too heavy. Which statement is correct?
Try an exam question
A school survey asks students: "Don't you agree that homework is far too long?" Comment on the question and suggest an improvement.
[4 marks]
- [1]It is a leading (biased) question that pushes the respondent towards agreeing.
- [1]It offers no balanced scale or choices, so answers are hard to compare.
- [1]Improved wording is neutral, for example "How would you rate the amount of homework?".
- [1]Give consistent, non-overlapping choices, for example a scale from "far too little" to "far too much".
That's the notes covered.
Carry on to the next subtopic.
Try a question
What is the main weakness of the proposed question?
1 mark, marked against the mark scheme
What is the main weakness of the proposed question?
1 mark, marked against the mark scheme