Estimators, standard error and biasEdexcel A-Level Further Maths: Revision notes

A population parameter such as $\mu$ or $\sigma^2$ is a fixed but usually unknown number. An estimator is a statistic, a function of the random sample $X_1,\ldots,X_n$, used to estimate it, for example $\bar X=\frac1n\sum X_i$. An estimate is the numerical value the estimator takes for one observed sample. Because the sample is random, an estimator is itself a random variable with a sampling distribution, and its mean and variance describe how good it is.

The bias of an estimator $T$ of a parameter $\theta$ is $\mathrm{E}(T)-\theta$. $T$ is unbiased if $\mathrm{E}(T)=\theta$, so that on average it neither overestimates nor underestimates. A positive bias means $T$ overestimates on average; a negative bias means it underestimates. Two results to know: $\mathrm{E}(\bar X)=\mu$, so the sample mean is an unbiased estimate of $\mu$; and $S^2=\frac{1}{n-1}\sum(X_i-\bar X)^2$ is an unbiased estimate of $\sigma^2$. To test any linear estimator, use $\mathrm{E}(aX+bY)=a\mathrm{E}(X)+b\mathrm{E}(Y)$. For $T=\frac{X_1+2X_2+3X_3}{6}$: $\mathrm{E}(T)=\frac{\mu+2\mu+3\mu}{6}=\mu$, so $T$ is unbiased.

The sample variance is $s^2=\frac{1}{n-1}\sum(x_i-\bar x)^2=\frac{1}{n-1}\left(\sum x_i^2-\frac{(\sum x_i)^2}{n}\right)$. Deviations are measured from $\bar x$, which is fitted to the same data, so they are slightly too small. Dividing by $n$ gives $V$ with $\mathrm{E}(V)=\frac{n-1}{n}\sigma^2$, which underestimates $\sigma^2$ with bias $-\frac{\sigma^2}{n}$. Dividing by $n-1$ removes that bias exactly. Example: $n=10$, $\sum x=124.0$, $\sum x^2=1582.6$ gives $S_{xx}=1582.6-\frac{124^2}{10}=45$ and $s^2=\frac{45}{9}=5$.

For independent observations, $\mathrm{Var}(aX+bY)=a^2\mathrm{Var}(X)+b^2\mathrm{Var}(Y)$. Hence $\mathrm{Var}(\bar X)=\frac{\sigma^2}{n}$. The standard error of an estimator is the standard deviation of its sampling distribution. For the sample mean it is $\frac{\sigma}{\sqrt n}$, estimated by $\frac{s}{\sqrt n}$ when $\sigma$ is unknown. Example: $s^2=5$, $n=10$ gives estimated standard error $\sqrt{\frac{5}{10}}=0.707$. Quadrupling the sample size to $40$ halves it to $0.354$. A smaller standard error means the estimator is more precise: its values cluster more tightly around the parameter.

A good estimator is unbiased and has a small variance. To compare two estimators of the same parameter:

• If both are unbiased, prefer the one with the smaller variance (it is more efficient).
• A biased estimator may have a smaller variance but misses the target systematically, so it is usually rejected in favour of an unbiased one. Example: for $\mu$ from $n=4$, $A=\bar X$ has variance $\frac{\sigma^2}{4}$ and $B=\frac{X_1+X_2}{2}$ has variance $\frac{\sigma^2}{2}$. Both are unbiased, so $A$ is better, because $B$ throws away two observations. In an evaluation, state bias first, then variances, then a conclusion.