Estimators, standard error and bias

Estimators, standard error and bias

Spec 5.1 · Edexcel A-Level Further Maths: Revision notes

Studying this course? Keep your progress in one place.

Key facts

  • An estimator is a random statistic; an estimate is the number from one sample.
  • Bias =E(T)−θ=\mathrm{E}(T)-\theta. Xˉ\bar X is an unbiased estimator of μ\mu.
  • S2=1n−1∑(Xi−Xˉ)2S^2=\frac{1}{n-1}\sum(X_i-\bar X)^2 is unbiased for σ2\sigma^2; dividing by nn is biased low.
  • Standard error of Xˉ\bar X is σn\frac{\sigma}{\sqrt n}, estimated by sn\frac{s}{\sqrt n}.
  • Prefer the unbiased estimator with the smaller variance.

Estimators and estimates

An estimator is a random variable; an estimate is the number it gives for one sample.

A population parameter such as μ\mu or σ2\sigma^2 is a fixed, usually unknown number. An estimator is a statistic, a function of the random sample X1,…,XnX_1,\ldots,X_n, such as Xˉ\bar X. An estimate is the value it takes for one observed sample.

Because the sample is random, an estimator has a sampling distribution, whose mean and variance show how good it is.

  1. 1

    Population parameter

    fixed, unknown, e.g. μ\mu

  2. 2

    Random sample

    X1,…,XnX_1,\ldots,X_n

  3. 3

    Estimator

    a statistic such as Xˉ\bar X (a random variable)

  4. 4

    Estimate

    the number xˉ\bar x for one sample

From population to estimate

Which of these is a number, not a random variable?

Bias

An estimator is unbiased when its expected value equals the parameter.

The bias of an estimator TT of θ\theta is E(T)−θ\mathrm{E}(T)-\theta. TT is unbiased if E(T)=θ\mathrm{E}(T)=\theta. A positive bias means it overestimates on average; a negative bias, that it underestimates.

Know that E(Xˉ)=μ\mathrm{E}(\bar X)=\mu and that S2=1n−1∑(Xi−Xˉ)2S^2=\frac{1}{n-1}\sum(X_i-\bar X)^2 is unbiased for σ2\sigma^2. To test any linear estimator, use E(aX+bY)=aE(X)+bE(Y)\mathrm{E}(aX+bY)=a\mathrm{E}(X)+b\mathrm{E}(Y).

123456789100.10.20.30.40.5xyθUnbiased: E(T) = θBiased: E(T) > θ
Illustrative sampling distributions: an unbiased estimator is centred on the parameter θ = 5; a biased one is centred away from it (dashed).
  • BiasE(T)−θ\mathrm{E}(T)-\theta
  • UnbiasedE(T)=θ\mathrm{E}(T)=\theta

Worked example

Show that T=X1+2X2+3X36T=\frac{X_1+2X_2+3X_3}{6} is an unbiased estimator of μ\mu.

An estimator has E(T)=θ+2\mathrm{E}(T)=\theta+2. What is its bias?

Why divide by n − 1

Deviations from xˉ\bar x are slightly too small, and dividing by n−1n-1 corrects the bias exactly.

s2=1n−1∑(xi−xˉ)2=1n−1(∑xi2−(∑xi)2n)s^2=\frac{1}{n-1}\sum(x_i-\bar x)^2=\frac{1}{n-1}\left(\sum x_i^2-\frac{(\sum x_i)^2}{n}\right)

Deviations are measured from xˉ\bar x, which is fitted to the same data, so they are slightly too small. Dividing by nn gives VV with E(V)=n−1nσ2\mathrm{E}(V)=\frac{n-1}{n}\sigma^2, with bias −σ2n-\frac{\sigma^2}{n}.

00.20.40.60.81251020Sample size nE(estimator) / σ²
  • Divide by n
  • Divide by n − 1
E(V)/σ² when dividing by n is (n − 1)/n; dividing by n − 1 always gives 1
  • Unbiased estimate of sigma2sigma^2s2=Sxxn−1s^2=\dfrac{S_{xx}}{n-1}
  • Dividing by nnbiased low: E(V)=n−1nσ2\mathrm{E}(V)=\frac{n-1}{n}\sigma^2

Worked example

n=10n=10, ∑x=124.0\sum x=124.0, ∑x2=1582.6\sum x^2=1582.6. Find s2s^2.

Dividing ∑(xi−xˉ)2\sum(x_i-\bar x)^2 by nn gives an estimator of σ2\sigma^2 that is...

Standard error

The standard error is the standard deviation of the sampling distribution; for Xˉ\bar X it is σn\frac{\sigma}{\sqrt n}.

For independent observations, Var(aX+bY)=a2Var(X)+b2Var(Y)\mathrm{Var}(aX+bY)=a^2\mathrm{Var}(X)+b^2\mathrm{Var}(Y), so Var(Xˉ)=σ2n\mathrm{Var}(\bar X)=\frac{\sigma^2}{n}.

The standard error of Xˉ\bar X is σn\frac{\sigma}{\sqrt n}, estimated by sn\frac{s}{\sqrt n} when σ\sigma is unknown. A smaller standard error means a more precise estimator.

2468101214161820222412345xyσ/√n
Standard error σ/√n falls as the sample size n grows (x is n). Change σ.
  • Standard error of Xˉ\bar Xσn\dfrac{\sigma}{\sqrt n}, estimated by sn\dfrac{s}{\sqrt n}

Worked example

s2=5s^2=5, n=10n=10. Find the estimated standard error, then the value for n=40n=40.

The sample size is multiplied by 99. The standard error is...

Choosing between estimators

Prefer the unbiased estimator with the smaller variance, which is more efficient.

A good estimator is unbiased and has a small variance. If both are unbiased, prefer the one with the smaller variance: it is more efficient.

A biased estimator may have a smaller variance but misses the target systematically, so it is usually rejected. In an evaluation, state bias first, then variances, then a conclusion.

123456789100.10.20.30.40.5xyA: variance σ²/4B: variance σ²/2
Sampling distributions of two unbiased estimators of θ = 5 (σ = 2, illustrative): A = X̄ with n = 4 has variance 1; B has variance 2. A is more efficient.

A = X̄ (n = 4)

Bias:
Unbiased
Variance:
σ²/4

B = (X₁ + X₂)/2

Bias:
Unbiased
Variance:
σ²/2

Worked example

For μ\mu from n=4n=4, compare A=XˉA=\bar X and B=X1+X22B=\frac{X_1+X_2}{2}.

Two unbiased estimators of the same parameter have different variances. Which is preferred?

Try an exam question

X1,X2,X3X_1,X_2,X_3 is a random sample from a population with mean μ\mu and variance σ2\sigma^2. The estimator U=2X1+X23U=\frac{2X_1+X_2}{3} is used to estimate μ\mu. Show that UU is unbiased, find Var(U)\mathrm{Var}(U), and state with a reason whether UU or Xˉ\bar X is the better estimator.

[4 marks]

That's the notes covered.

Carry on to the next subtopic.

Try a question

Which statement about UU is correct?

1 mark, marked against the mark scheme

Revision notes