4.12 Data collection, reliability and validityIB Maths: Applications and Interpretation HL: Revision notes

A survey or questionnaire collects data from a sample. A valid method begins with the question to be answered: choose the relevant variables from the many that could be recorded, and collect relevant and appropriate data. Collecting everything makes the analysis messy; collecting too little makes a conclusion impossible. The sample must represent the population; asking only people who are easy to reach (for example students leaving the canteen) can introduce bias. Check that the data type suits the analysis: numerical data for means and regression, categorical data for $\chi^2$ tests.

Questions can be biased or unbiased. A biased (leading or loaded) question pushes the answer, for example "Don't you agree that the new menu is far better?". An unbiased question is neutral: "How satisfied are you with the new menu?". Personal questions (age, income, health) may get untruthful answers, so ask only what is needed and consider ranges. Unstructured (open) questions allow free answers; they give rich detail but are hard to analyse. Structured questions give consistent answer choices, for example a five-point scale from "very satisfied" to "very dissatisfied", which are easy to count and compare. The choices should be balanced, cover all possible answers and not overlap. Precise questioning means exact wording and units, for example "How many hours did you sleep last night?" rather than "Do you sleep well?".

To use a $\chi^2$ test on numerical data (for example hours slept), group the values into categories (classes) and count how many fall in each. Choose categories that make sense for the question and justify the choice, for example equal width, or boundaries with a natural meaning. The test requires expected frequencies greater than 5 in every category (a common rule). If a category has a smaller expected frequency, combine it with a neighbouring category and recalculate. In a contingency table the expected frequency is $\frac{\text{row total}\times\text{column total}}{\text{grand total}}$. Example: for "under 5 hours" with 6 students, expected alert $=\frac{6\times87}{155}=3.37<5$, so combine "under 5" with "5 to under 6".

For a $\chi^2$ goodness of fit test, with $k$ categories (after combining) the degrees of freedom are $\nu=k-1-m,$ where $m$ is the number of parameters estimated from the data (for example $p$ for a binomial, or the mean for a Poisson). If the parameters are given, $m=0$. For a contingency table, $\nu=(\text{rows}-1)(\text{columns}-1)$. Example: a binomial $B(5,p)$ with $p$ estimated, and 3 categories after combining, gives $\nu=3-1-1=1$. Use your GDC for $\chi^2$ and the $p$-value: reject $H_0$ if $p<$ the significance level.

A test is reliable if it gives consistent results when repeated under the same conditions. Reliability does not depend on whether the test measures the right thing. Test-retest: give the same test to the same people on two occasions and find the correlation between the scores; a high $r$ (close to 1) means high reliability. Parallel forms: give two different but equivalent versions of the test to the same people and correlate the scores.

A test is valid if it measures what it is intended to measure. Content validity: the questions cover all the relevant content (for example an algebra test that includes all algebra topics in the syllabus), usually judged by experts. Criterion-related validity: scores agree with an independent, accepted measure of the same thing (the criterion), for example a questionnaire compared with a counsellor's diagnosis. The two ideas are different: a bathroom scale that is always 3 kg too heavy is reliable (consistent) but not valid (accurate). A test can be reliable without being valid, but a test that is not reliable cannot be valid.