4.12 Data collection, reliability and validity

4.12 Data collection, reliability and validity

IB Maths: Applications and Interpretation HL: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • Choose relevant variables and a sample that represents the population, or the data will be biased.
  • Questions should be neutral, precise and give consistent answer choices.
  • For a χ2\chi^2 test, expected frequencies must be above 55: combine categories if not.
  • Goodness of fit has ν=k−1−m\nu=k-1-m, where mm is the number of parameters estimated from the data.
  • Reliable means consistent; valid means measures the right thing. Reliable does not mean valid.

Designing a valid method

Start from the question: collect relevant data from a sample that represents the population.

A survey or questionnaire collects data from a sample. Choose the relevant variables from the many that could be recorded: collecting everything makes analysis messy, collecting too little makes a conclusion impossible.

The sample must represent the population. Asking only people who are easy to reach (students leaving the canteen) can introduce bias.

Check the data type suits the analysis: numerical data for means and regression, categorical data for χ2\chi^2 tests.

  1. 1

    Question

    What must the data answer?

  2. 2

    Variables

    Record only the relevant ones

  3. 3

    Sample

    Representative, not just convenient

  4. 4

    Data type

    Suits the analysis planned

Planning a survey

A survey about school meals only asks students leaving the canteen. What is the main problem?

Writing good questions

Good questions are neutral, precise and offer balanced answer choices.

A biased question leads the respondent: "Don't you agree that the new menu is far better?". A neutral one asks "How satisfied are you with the new menu?".

Personal questions (age, income, health) may get untruthful answers, so ask only what is needed and consider ranges.

Structured questions give consistent choices, such as a five-point scale; they are easy to count. Unstructured questions allow free answers: rich but hard to analyse. Choices should cover every answer and not overlap.

Precise wording means exact units: "How many hours did you sleep last night?".

11.522.533.544.551 Very dissatisfied2 Dissatisfied3 Neutral4 Satisfied5 Very satisfied

Biased

  • Don't you agree the canteen is far too expensive?
  • Surely the prices are unfair?

Unbiased

  • How would you rate the canteen prices from very low to very high?
  • How satisfied are you with the new menu?

Structured

Advantage:
Easy to count and compare
Disadvantage:
May not include the respondent's answer

Unstructured

Advantage:
Rich, detailed answers
Disadvantage:
Hard to analyse

Which question is unbiased?

Categories for a chi-squared test

Group the values into justified categories, and combine categories until every expected frequency is above 55.

To use a χ2\chi^2 test on numerical data (for example hours slept), sort the values into categories and count each. Choose categories that make sense and justify the choice, for example equal widths.

The test needs expected frequencies above 55 in every category. If one is smaller, combine categories with a neighbour and recalculate.

The rule applies to the expected values, not the observed ones.

0246810121416Under 55 to 66 to 77 to 88 to 9Over 9Hours sleptExpected frequency
Illustrative expected frequencies for hours slept. The first and last categories fall below 5, so each is combined with its neighbour.
  • Expected frequencyrow total×column totalgrand total\dfrac{\text{row total}\times\text{column total}}{\text{grand total}}
  • Rule of 5every expected frequency >5>5

Worked example

In a table of 155155 students, the "under 5 hours" row total is 66 and the "alert" column total is 8787. Is the expected frequency large enough?

Which values must be greater than 55 for the χ2\chi^2 test?

Degrees of freedom with estimates

Subtract 11 for the total and 11 more for each parameter estimated from the data.

For a χ2\chi^2 goodness of fit test with kk categories (after combining), ν=k−1−m\nu=k-1-m, where mm is the number of parameters estimated from the data, such as pp for a binomial or the mean for a Poisson. If the parameters are given, m=0m=0.

For a contingency table, ν=(rows−1)(columns−1)\nu=(\text{rows}-1)(\text{columns}-1).

Use your GDC for χ2\chi^2 and the pp-value, and reject H0H_0 if p<p< the significance level.

024681012123456Degrees of freedom νCritical value at 5%
The 5% critical value of χ² grows with the degrees of freedom ν.
  • Goodness of fitν=k−1−m\nu=k-1-m
  • Contingency tableν=(r−1)(c−1)\nu=(r-1)(c-1)

Worked example

A binomial B(5,p)B(5,p) is fitted with pp estimated from the data. After combining there are 33 categories. Find ν\nu.

A Poisson model with the mean estimated from the data is tested with 55 categories. Find ν\nu.

Reliability

A test is reliable if it gives consistent results when repeated under the same conditions.

Reliable says nothing about whether the test measures the right thing.

Test-retest: give the same test to the same people twice and find the correlation between the scores. A value of rr close to 11 means high reliability.

Parallel forms: give two different but equivalent versions to the same people and correlate the scores.

  1. 1

    First sitting

    Give the test to a group

  2. 2

    Second sitting

    Same test, same people, same conditions

  3. 3

    Correlate

    Find r between the two sets of scores

  4. 4

    Interpret

    r close to 1 means high reliability

Test-retest reliability.

Test-retest

  • Same test, same people, twice
  • Correlate the two sets of scores

Parallel forms

  • Two equivalent versions, same people
  • Correlate the two sets of scores

A test-retest correlation is r=0.95r=0.95. What does this show?

Validity versus reliability

A test is valid if it measures what it is meant to measure; it can be reliable without being valid.

Content validity: the questions cover all the relevant content, usually judged by experts. Criterion-related validity: scores agree with an independent, accepted measure of the same thing.

A bathroom scale that is always 33 kg too heavy is reliable (consistent) but not valid (accurate). A test that is not reliable cannot be valid.

O1O2a1a2a3b1b2b3
Target analogy: left, reliable and valid (tight cluster on the centre); right, reliable but not valid (tight cluster, off target).

Reliable

Question it answers:
Are the results consistent?
Example of failure:
Scores change a lot between two sittings

Valid

Question it answers:
Does it measure the right thing?
Example of failure:
A scale always 3 kg too heavy

A scale always reads 3 kg too heavy. Which statement is correct?

Try an exam question

A school survey asks students: "Don't you agree that homework is far too long?" Comment on the question and suggest an improvement.

[4 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

What is the main weakness of the proposed question?

1 mark, marked against the mark scheme

Revision notes