Estimators, bias and variance

An estimator is a recipe that turns a sample into a guess. Two recipes can target the same quantity and still behave very differently from sample to sample. This chapter measures that behaviour the way the 2020 notebook did — by simulation — and compares it with the theory.

§1.1Coursework A1 Q4

Midrange or mean?

With a sample of 20 from a normal distribution, one suggestion was to estimate μ by the midrange, the average of the smallest and largest values; the other was to use the sample mean. Both are unbiased by symmetry, so the question is spread. The notebook drew 100 samples with set.seed(2529) and compared

\operatorname{bias}(T)=\mathbb E(T)-\mu,

\operatorname{var}(T)=\mathbb E\big[(T-\mathbb E T)^2\big].

The port re-implements R’s random number generator, so the default settings below replay those same 2,000 normal draws (identical to the last bit or two) and print the 2020 results exactly.

Random seed (R’s set.seed)

Sample size n20

Simulated samples100

Above about 60 000 draws the simulation runs in a Web Worker.

These are the exact draws from the 2020 notebook: set.seed(2529), 100 samples of rnorm(20).

Midrange bias

0.043088

Midrange variance

0.158578

Mean bias

-0.020101

Mean variance

0.069188

theory σ²/n = 0.05

Relative efficiency var(midrange) / var(mean)2.2995% paired bootstrap interval 1.76 to 3.05 (B = 2,000, seed 1), identical in R

Midrange (min + max) / 2
Sample mean
True mean μ = 0

Figure 1.1.Sampling distributions of the two estimators over 100 samples of size 20 from N(0, 1). Both are centred on μ, but the midrange is 2.29× as variable: it relies on the two most extreme observations, while the mean uses every one.

Show the numbers as a table

estimator	bias	variance	MSE	median	IQR (hinges)
Midrange	0.043088	0.158578	0.158849	0.0268	0.6123
Sample mean	-0.020101	0.069188	0.068900	-0.0159	0.3598

§1.2Coursework A1 Q3

Method of moments against maximum likelihood

For $X_1,\dots,X_n \sim \mathrm{Unif}(0,\theta)$ the method of moments matches $\;\mathbb E X=\theta/2\;$ to the sample mean, while the likelihood $\;L(\theta)=\theta^{-n}\;$ for $\theta\ge\max X_i$ is maximised at the largest observation:

\hat\theta_{\text{MME}} = 2\bar X,

\hat\theta_{\text{MLE}} = X_{(n)} .

Mean squared error splits into variance plus squared bias, $\;\mathrm{MSE}=\operatorname{var}(\hat\theta)+\operatorname{bias}(\hat\theta)^2$ , giving $\theta^2/(3n)$ for the MME and $2\theta^2/\{(n+1)(n+2)\}$ for the MLE. Multiplying the MLE by $a=(n+2)/(n+1)$ minimises MSE among multiples of it.

Random seed (R’s set.seed)

Sample size n10

True endpoint θ3.0

Simulated samples2000

Try n = 1: a single observation, as in part (a) of the question. The MLE is then just X itself, biased downwards by half.

Method of moments 2X̄bias 0.030 · var 0.2876 · MSE 0.2884 ± 0.0091 (theory 0.3000)

Maximum likelihood max(Xᵢ)bias -0.263 · var 0.0577 · MSE 0.1270 ± 0.0052 (theory 0.1364)

Scaled MLE 1.091·max(Xᵢ)bias -0.014 · var 0.0687 · MSE 0.0689 ± 0.0034 (theory 0.0744)

Solid black line: the true θ. Dashed line: the average of the simulated estimates. The ± after each simulated MSE is its Monte Carlo standard error, sd((θ̂ − θ)²) / √R.

Figure 1.2.Sampling distributions of three estimators of θ = 3 from samples of size 10 (2,000 simulations each, shared x axis). The MME is unbiased but spread out; the MLE is tightly packed but always below θ; rescaling the MLE trades a sliver of variance for zero bias and wins on mean squared error.

Show simulated and theoretical moments

estimator	mean (sim)	mean (theory)	variance (sim)	MSE (sim)	± MCSE	MSE (theory)
Method of moments 2X̄	3.0300	3.0000	0.28760	0.28835	0.00906	0.30000
Maximum likelihood max(Xᵢ)	2.7367	2.7273	0.05774	0.12703	0.00517	0.13636
Scaled MLE 1.091·max(Xᵢ)	2.9855	3.0000	0.06872	0.06889	0.00345	0.07438

§1.3Coursework A3 Q3

A Bayesian estimate

Assignment 3 swapped the frequentist view for a prior. With data from $f(x\mid\beta)=\beta^2 x e^{-\beta x}$ and the prior $f(\beta)=e^{-\beta}$ , the posterior is proportional to $\beta^{2n}e^{-\beta(\sum x_i+1)}$ , which is

\beta \mid x \sim \mathrm{Gamma}\Big(2n+1,\ \textstyle\sum_{i} x_i+1\Big),

\mathbb E(\beta\mid x)=\frac{2n+1}{\sum x_i+1},

\operatorname{sd}(\beta\mid x)=\frac{\sqrt{2n+1}}{\sum x_i+1}.

Random seed (R’s set.seed)

True β1.5

Observations n10

Data are drawn from f(x | β) = β²x e^−βx, a Gamma(2, β) distribution, as the sum of two exponential draws. Set n = 0 to see the prior on its own.

Posterior

Γ(21, 12.77)

shape 2n + 1, rate Σx + 1

Posterior mean

1.645

sd 0.359

95% credible interval

(1.02, 2.42)

MLE 2n / Σx

1.699

true β = 1.5

Prior e^(−β)
Posterior
MLE
True β

Figure 1.3.The Exp(1) prior is conjugate here: multiplying it by the likelihood β²ⁿe^−βΣx leaves another gamma density. With 10 observations the posterior mean (2n + 1)/(Σx + 1) sits between the prior mean 1 and the MLE, and the posterior narrows like 1/√n.