Skip to content
Statistics LabMAST20005 · 2020

Statistics Lab / Appendix

Simulation studies

The 2020 answers applied each method once, to one dataset. These studies apply five of them thousands of times to data whose truth is known, and count how often an interval covers or a test rejects. Every rate comes with a Wilson 95% interval and a Monte Carlo standard error, so the simulations are held to the same standard as the statistics they check.

5 studies · 49 cells
design declared in code
245,000 replications
5 seed blocks × 1,000 per cell
51 of 51 exact values
inside their Wilson 95% intervals
Every count replayed in R
scripts/r_studies.R, identical

How to read these studies

Design, seeds and uncertainty

Each cell is one method under one data-generating setting. It runs five independent blocks of 1,000 replications, and block k of cell c starts from set.seed(base + 100c + k), so the seeds are on the page and anyone can rerun a block in R.

A rate of 95% from 5,000 replications has a Monte Carlo standard error of about 0.3 percentage points, p(1−p)/R\sqrt{p(1-p)/R}, and a Wilson interval about ±0.6 points wide. Where theory gives the exact rate it is marked ◇, and it should fall inside the interval about 19 times in 20.

When two methods are compared they run on the same simulated datasets. The difference is reported with Newcombe’s paired interval and an exact McNemar test, which uses the pairing instead of comparing two separate intervals.

§B.1Coursework A2 Q2(a)

Does the t interval keep its 95%?

The 2020 interval for a mean walking time, xˉ±tn−1 s/n\bar x \pm t_{n-1}\,s/\sqrt n, is exact when the data are normal, because (Xˉ−μ)/(S/n)(\bar X-\mu)/(S/\sqrt n) then has a t distribution. Real times are often skewed. This study draws samples from a normal, an exponential and a lognormal distribution and checks how often the interval catches the true mean.

Committed resultscommitted, replayed in R

12 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(20005 + 100c + k).

Replications per block (study B.1)

  • Normal(15, 3²)
  • Exponential, mean 15
  • Lognormal(0, 1)
  • Exact (normal data)
  • Nominal 95%
Figure B.1.Coverage of the 95% t interval for a mean, 5,000 samples per point, with Wilson 95% intervals. Under normal data it holds at every n. For skewed data it falls short, most of all at small n, and the misses are almost all on one side (table below).
Coverage of the t interval
datancoverage [Wilson 95%] ◇ exactinterval below μinterval above μMCSE
Normal(15, 3²)595.2% [94.6–95.8]◇ 95.00%2.4%2.4%0.30%
Normal(15, 3²)1095.0% [94.4–95.6]◇ 95.00%2.2%2.8%0.31%
Normal(15, 3²)2095.0% [94.4–95.6]◇ 95.00%2.3%2.7%0.31%
Normal(15, 3²)5094.6% [93.9–95.2]◇ 95.00%2.7%2.7%0.32%
Exponential, mean 15589.1% [88.2–89.9]10.4%0.5%0.44%
Exponential, mean 151090.2% [89.4–91.0]9.3%0.5%0.42%
Exponential, mean 152091.8% [91.0–92.5]7.8%0.5%0.39%
Exponential, mean 155093.2% [92.5–93.9]5.9%0.9%0.36%
Lognormal(0, 1)581.9% [80.8–82.9]18.0%0.1%0.54%
Lognormal(0, 1)1084.1% [83.1–85.1]15.8%0.1%0.52%
Lognormal(0, 1)2087.4% [86.5–88.3]12.5%0.1%0.47%
Lognormal(0, 1)5089.8% [88.9–90.6]9.9%0.3%0.43%

Every rate above is derived from these per-seed counts.

Per-seed counts and the spread across seeds

Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.

Per-seed counts for every cell and outcome
celloutcomeseedsper-seed countssd / binomial
t-normal-n5cover20005…20009960 · 949 · 959 · 941 · 9531.16
t-normal-n5below20005…2000918 · 29 · 22 · 27 · 240.89
t-normal-n5above20005…2000922 · 22 · 19 · 32 · 231.03
t-normal-n10cover20105…20109950 · 951 · 963 · 943 · 9451.14
t-normal-n10below20105…2010923 · 25 · 16 · 24 · 220.76
t-normal-n10above20105…2010927 · 24 · 21 · 33 · 331.04
t-normal-n20cover20205…20209949 · 940 · 960 · 947 · 9551.11
t-normal-n20below20205…2020918 · 29 · 22 · 23 · 240.83
t-normal-n20above20205…2020933 · 31 · 18 · 30 · 211.31
t-normal-n50cover20305…20309952 · 944 · 951 · 941 · 9420.72
t-normal-n50below20305…2030921 · 28 · 27 · 29 · 300.69
t-normal-n50above20305…2030927 · 28 · 22 · 30 · 280.59
t-exponential-n5cover20405…20409887 · 892 · 883 · 912 · 8801.29
t-exponential-n5below20405…20409107 · 102 · 112 · 85 · 1141.20
t-exponential-n5above20405…204096 · 6 · 5 · 3 · 60.57
t-exponential-n10cover20505…20509897 · 891 · 907 · 906 · 9100.84
t-exponential-n10below20505…2050996 · 104 · 90 · 86 · 890.78
t-exponential-n10above20505…205097 · 5 · 3 · 8 · 11.31
t-exponential-n20cover20605…20609911 · 898 · 919 · 933 · 9281.60
t-exponential-n20below20605…2060979 · 97 · 77 · 65 · 701.44
t-exponential-n20above20605…2060910 · 5 · 4 · 2 · 21.54
t-exponential-n50cover20705…20709941 · 949 · 934 · 923 · 9141.76
t-exponential-n50below20705…2070948 · 47 · 54 · 68 · 761.73
t-exponential-n50above20705…2070911 · 4 · 12 · 9 · 101.03
t-lognormal-n5cover20805…20809800 · 824 · 823 · 818 · 8290.92
t-lognormal-n5below20805…20809199 · 176 · 176 · 181 · 1700.91
t-lognormal-n5above20805…208091 · 0 · 1 · 1 · 10.50
t-lognormal-n10cover20905…20909829 · 837 · 839 · 856 · 8460.88
t-lognormal-n10below20905…20909171 · 163 · 160 · 142 · 1530.95
t-lognormal-n10above20905…209090 · 0 · 1 · 2 · 10.94
t-lognormal-n20cover21005…21009878 · 873 · 862 · 886 · 8710.84
t-lognormal-n20below21005…21009122 · 126 · 137 · 113 · 1280.84
t-lognormal-n20above21005…210090 · 1 · 1 · 1 · 10.50
t-lognormal-n50cover21105…21109888 · 902 · 898 · 905 · 8960.68
t-lognormal-n50below21105…21109108 · 95 · 101 · 94 · 990.59
t-lognormal-n50above21105…211094 · 3 · 1 · 1 · 51.07

Explain this resultoptional · your own key

Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.

§B.2Coursework A1 Q2(c)

The lognormal variance interval, as submitted

The 2020 code estimated λ = σ² of log X by its MLE, λ^=SS/n\hat\lambda = SS/n, and then used the pivot built for the unbiased S2=SS/(n−1)S^2 = SS/(n-1):

((n−1)λ^χn−1, 0.9752, (n−1)λ^χn−1, 0.0252)\left(\frac{(n-1)\hat\lambda}{\chi^2_{n-1,\,0.975}},\ \frac{(n-1)\hat\lambda}{\chi^2_{n-1,\,0.025}}\right)
=n−1n(SSχn−1, 0.9752, SSχn−1, 0.0252).= \frac{n-1}{n}\left(\frac{SS}{\chi^2_{n-1,\,0.975}},\ \frac{SS}{\chi^2_{n-1,\,0.025}}\right).

The result is the textbook interval shrunk by (n − 1)/n. Its exact behaviour follows from SS/λ∼χn−12SS/\lambda\sim\chi^2_{n-1}, and the simulation checks it on samples drawn under the lognormal model.

Committed resultscommitted, replayed in R

3 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(1520 + 100c + k).

Replications per block (study B.2)

  • 2020 interval lies below λ
  • 2020 interval lies above λ
  • Exact, from χ²
  • 2.5% each side if equal-tailed
Figure B.2.How the 2020 interval misses. Overall coverage stays near 95%, but the interval sits too low. It lies entirely below λ between 1.6 and 5 times as often as it lies above it, while the textbook interval misses 2.5% on each side.
Coverage of the two lambda intervals
n2020 interval [Wilson] ◇ exactbelow / above λ (exact)textbook SS/χ² [Wilson] ◇ exact2020 − textbook, points
595.5% [94.9–96.0]◇ 95.49%3.76% / 0.75%95.2% [94.6–95.7]◇ 95.00%0.3 (−0.2 to 0.8)
1594.9% [94.2–95.4]◇ 95.14%3.43% / 1.43%94.9% [94.2–95.4]◇ 95.00%0.0 (−0.4 to 0.4)
5094.6% [93.9–95.2]◇ 95.04%3.05% / 1.91%94.4% [93.8–95.0]◇ 95.00%0.1 (−0.2 to 0.4)

Every rate above is derived from these per-seed counts.

Per-seed counts and the spread across seeds

Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.

Per-seed counts for every cell and outcome
celloutcomeseedsper-seed countssd / binomial
lambda-n5submitted1520…1524951 · 954 · 968 · 950 · 9521.13
lambda-n5submittedBelow1520…152441 · 33 · 25 · 44 · 381.26
lambda-n5submittedAbove1520…15248 · 13 · 7 · 6 · 100.94
lambda-n5textbook1520…1524950 · 950 · 965 · 949 · 9451.13
lambda-n5textbookBelow1520…152425 · 22 · 15 · 25 · 240.90
lambda-n5textbookAbove1520…152425 · 28 · 20 · 26 · 310.81
lambda-n15submitted1620…1624950 · 944 · 942 · 949 · 9580.89
lambda-n15submittedBelow1620…162436 · 41 · 45 · 36 · 241.33
lambda-n15submittedAbove1620…162414 · 15 · 13 · 15 · 180.49
lambda-n15textbook1620…1624946 · 940 · 947 · 951 · 9591.01
lambda-n15textbookBelow1620…162426 · 25 · 33 · 23 · 151.33
lambda-n15textbookAbove1620…162428 · 35 · 20 · 26 · 261.05
lambda-n50submitted1720…1724938 · 947 · 958 · 946 · 9401.09
lambda-n50submittedBelow1720…172439 · 28 · 18 · 38 · 371.61
lambda-n50submittedAbove1720…172423 · 25 · 24 · 16 · 230.76
lambda-n50textbook1720…1724936 · 945 · 952 · 948 · 9410.85
lambda-n50textbookBelow1720…172433 · 25 · 16 · 30 · 311.33
lambda-n50textbookAbove1720…172431 · 30 · 32 · 22 · 280.75

Explain this resultoptional · your own key

Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.

§B.3Coursework A2 Q1

Wald or Agresti–Caffo for two proportions?

The council survey compared 520 of 800 with 600 of 1,000 using the Wald interval p^1−p^2±z se^\hat p_1-\hat p_2\pm z\,\widehat{\mathrm{se}}. The Agresti–Caffo interval adds one success and one failure to each group before using the same formula. Exact coverage comes from summing binomial probabilities over every possible pair of counts, and the simulation runs both intervals on the same surveys.

Committed resultscommitted, replayed in R

3 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(800 + 100c + k).

Replications per block (study B.3)

  • Wald (2020)
  • Agresti–Caffo
  • Exact, by enumeration
  • Nominal 95%
Figure B.3.Coverage of two 95% intervals for p₁ − p₂ with p₁ = 0.65 and p₂ = 0.60, on the same 5,000 simulated surveys per size. At the coursework’s 800 and 1,000 they agree. In small groups the Wald interval falls short.
Coverage of the two-proportion intervals
groupsWald [Wilson] ◇ exactAgresti–Caffo [Wilson] ◇ exactAC − Wald, points (Newcombe)surveys where they disagree
16, 2094.0% [93.3–94.6]◇ 93.83%95.6% [95.0–96.1]◇ 95.58%1.6 (1.3 to 2.0)80
40, 5094.5% [93.9–95.1]◇ 94.42%95.2% [94.6–95.8]◇ 95.13%0.7 (0.4 to 0.9)33
800, 100094.8% [94.2–95.4]◇ 94.98%94.8% [94.2–95.4]◇ 94.99%0.0 (−0.1 to 0.1)2

Every rate above is derived from these per-seed counts.

Per-seed counts and the spread across seeds

Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.

Per-seed counts for every cell and outcome
celloutcomeseedsper-seed countssd / binomial
prop-16-20wald800…804947 · 940 · 927 · 939 · 9461.06
prop-16-20ac800…804959 · 958 · 941 · 963 · 9581.31
prop-40-50wald900…904948 · 939 · 955 · 943 · 9420.87
prop-40-50ac900…904951 · 948 · 959 · 951 · 9510.61
prop-800-1000wald1000…1004944 · 938 · 952 · 958 · 9491.09
prop-800-1000ac1000…1004944 · 938 · 951 · 959 · 9491.12

Explain this resultoptional · your own key

Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.

§B.4Coursework A2 Q2(c), A3 Q1(b)

Pooled t or Wilcoxon?

The coursework compared two groups twice, once with a pooled t interval and once with a Wilcoxon rank-sum test on two weeks of 7 days. This study runs both tests at α = 5% on the same datasets, for normal, heavy-tailed and skewed data, shifting the second group by 0 to 1.5 standard deviations.

With 7 per group the exact Wilcoxon test cannot reject at exactly 5%. Its largest rejection region below 5% has probability 3.79%, its attainable size.

Committed resultscommitted, replayed in R

24 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(3000 + 100c + k).

Replications per block (study B.4)

Observations per group

  • Pooled t test
  • Wilcoxon rank-sum (exact)
  • α = 5%

Normal

t, 3 df (heavy tails)

Exponential (skewed)

Figure B.4.Rejection rate against the size of the location shift, in standard deviations, for 7 observations per group. At a shift of 0 the rate is the test’s size. Both tests see the same 5,000 datasets in each cell.
Size and power, 7 per group
datashiftpooled t [Wilson]Wilcoxon [Wilson] ◇ exact sizeW − t, points (Newcombe)McNemar p
Normal05.1% [4.5–5.7]3.8% [3.3–4.3]◇ 3.79%−1.3 (−1.7 to −0.9)< 0.001
Normal0.513.8% [12.8–14.7]10.7% [9.9–11.6]−3.0 (−3.6 to −2.4)< 0.001
Normal140.8% [39.5–42.2]34.2% [32.9–35.5]−6.7 (−7.5 to −5.8)< 0.001
Normal1.573.5% [72.2–74.7]66.3% [65.0–67.6]−7.2 (−8.0 to −6.4)< 0.001
t, 3 df (heavy tails)04.0% [3.5–4.6]3.7% [3.2–4.3]◇ 3.79%−0.3 (−0.7 to 0.1)0.149
t, 3 df (heavy tails)0.519.8% [18.7–20.9]18.1% [17.1–19.2]−1.7 (−2.4 to −1.0)< 0.001
t, 3 df (heavy tails)155.4% [54.0–56.8]55.1% [53.7–56.5]−0.3 (−1.2 to 0.6)0.518
t, 3 df (heavy tails)1.582.6% [81.5–83.6]82.6% [81.5–83.6]0.0 (−0.8 to 0.8)0.961
Exponential (skewed)03.7% [3.2–4.2]3.4% [2.9–3.9]◇ 3.79%−0.3 (−0.8 to 0.1)0.167
Exponential (skewed)0.517.0% [16.0–18.0]19.9% [18.8–21.0]2.9 (2.1 to 3.7)< 0.001
Exponential (skewed)149.2% [47.8–50.6]50.7% [49.3–52.1]1.5 (0.6 to 2.5)0.002
Exponential (skewed)1.577.3% [76.1–78.5]77.1% [76.0–78.3]−0.2 (−1.1 to 0.7)0.732

Every rate above is derived from these per-seed counts.

Per-seed counts and the spread across seeds

Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.

Per-seed counts for every cell and outcome
celloutcomeseedsper-seed countssd / binomial
power-normal-n7-d0t3000…300455 · 50 · 48 · 49 · 520.40
power-normal-n7-d0wilcoxon3000…300445 · 33 · 41 · 32 · 380.90
power-normal-n7-d0.5t3100…3104140 · 136 · 130 · 136 · 1460.54
power-normal-n7-d0.5wilcoxon3100…3104111 · 109 · 105 · 107 · 1050.27
power-normal-n7-d1t3200…3204410 · 409 · 433 · 400 · 3891.04
power-normal-n7-d1wilcoxon3200…3204353 · 347 · 357 · 334 · 3171.09
power-normal-n7-d1.5t3300…3304731 · 750 · 731 · 722 · 7390.75
power-normal-n7-d1.5wilcoxon3300…3304653 · 674 · 654 · 651 · 6820.95
power-t3-n7-d0t3400…340440 · 39 · 42 · 41 · 400.18
power-t3-n7-d0wilcoxon3400…340436 · 41 · 34 · 41 · 340.60
power-t3-n7-d0.5t3500…3504194 · 195 · 204 · 182 · 2161.00
power-t3-n7-d0.5wilcoxon3500…3504187 · 170 · 184 · 169 · 1970.98
power-t3-n7-d1t3600…3604537 · 536 · 594 · 574 · 5291.81
power-t3-n7-d1wilcoxon3600…3604527 · 546 · 595 · 568 · 5181.99
power-t3-n7-d1.5t3700…3704838 · 820 · 809 · 829 · 8320.94
power-t3-n7-d1.5wilcoxon3700…3704835 · 818 · 818 · 839 · 8200.85
power-exponential-n7-d0t3800…380441 · 29 · 36 · 43 · 350.92
power-exponential-n7-d0wilcoxon3800…380436 · 27 · 33 · 38 · 340.73
power-exponential-n7-d0.5t3900…3904170 · 154 · 177 · 177 · 1710.79
power-exponential-n7-d0.5wilcoxon3900…3904198 · 191 · 200 · 206 · 2000.43
power-exponential-n7-d1t4000…4004497 · 506 · 487 · 509 · 4601.25
power-exponential-n7-d1wilcoxon4000…4004510 · 532 · 499 · 522 · 4731.44
power-exponential-n7-d1.5t4100…4104779 · 771 · 756 · 780 · 7800.78
power-exponential-n7-d1.5wilcoxon4100…4104767 · 782 · 759 · 779 · 7700.70
power-normal-n20-d0t4200…420448 · 57 · 61 · 38 · 451.35
power-normal-n20-d0wilcoxon4200…420443 · 48 · 59 · 46 · 450.93
power-normal-n20-d0.5t4300…4304352 · 344 · 332 · 353 · 3520.59
power-normal-n20-d0.5wilcoxon4300…4304336 · 331 · 319 · 329 · 3320.43
power-normal-n20-d1t4400…4404870 · 860 · 869 · 881 · 8881.04
power-normal-n20-d1wilcoxon4400…4404859 · 847 · 852 · 860 · 8630.59
power-normal-n20-d1.5t4500…4504999 · 998 · 998 · 994 · 9951.21
power-normal-n20-d1.5wilcoxon4500…4504993 · 997 · 999 · 993 · 9931.27
power-t3-n20-d0t4600…460449 · 43 · 34 · 63 · 561.65
power-t3-n20-d0wilcoxon4600…460455 · 42 · 47 · 67 · 611.41
power-t3-n20-d0.5t4700…4704422 · 442 · 464 · 441 · 4191.16
power-t3-n20-d0.5wilcoxon4700…4704562 · 571 · 537 · 542 · 5550.89
power-t3-n20-d1t4800…4804881 · 887 · 874 · 894 · 8910.80
power-t3-n20-d1wilcoxon4800…4804968 · 965 · 979 · 972 · 9691.00
power-t3-n20-d1.5t4900…4904982 · 988 · 987 · 975 · 9761.42
power-t3-n20-d1.5wilcoxon4900…49041000 · 1000 · 1000 · 998 · 9991.16
power-exponential-n20-d0t5000…500444 · 42 · 36 · 56 · 321.44
power-exponential-n20-d0wilcoxon5000…500454 · 40 · 38 · 53 · 461.10
power-exponential-n20-d0.5t5100…5104371 · 388 · 361 · 343 · 3801.15
power-exponential-n20-d0.5wilcoxon5100…5104576 · 594 · 595 · 564 · 5990.96
power-exponential-n20-d1t5200…5204883 · 876 · 861 · 865 · 8561.04
power-exponential-n20-d1wilcoxon5200…5204966 · 968 · 951 · 956 · 9591.14
power-exponential-n20-d1.5t5300…5304981 · 987 · 985 · 988 · 9911.01
power-exponential-n20-d1.5wilcoxon5300…5304996 · 999 · 1000 · 998 · 9981.11

Explain this resultoptional · your own key

Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.

§B.5Coursework A3 Q5

Checking the power curve of the rate test

For an exponential sample of n = 50, the 2020 answer tested λ = 1 with an equal-tailed version of the likelihood-ratio test: reject when Y=∑XiY=\sum X_i falls outside the 2.5% and 97.5% points of Gamma(50, 1). Its power at any λ then follows from Gamma(50, λ). Simulating the test directly is an independent check of that algebra.

Committed resultscommitted, replayed in R

7 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(5050 + 100c + k).

Replications per block (study B.5)

  • Analytic power from Gamma(50, λ)
  • Simulated, 5,000 samples each
  • α = 5%
Figure B.5.The power curve drawn in Chapter 4 comes from the gamma distribution of ΣXᵢ. In the committed run, the Wilson interval of every simulated rate contains the analytic value, which supports both the curve and the critical values a ≈ 37.11 and b ≈ 64.78.
Simulated and exact power of the 2020 equal-tailed likelihood-ratio test
true λsimulated [Wilson]exactinside interval
0.773.8% [72.6–75.0]73.64%yes
0.838.3% [37.0–39.7]38.18%yes
0.913.4% [12.5–14.4]12.71%yes
15.1% [4.5–5.7]5.00%yes
1.19.5% [8.7–10.3]9.36%yes
1.222.6% [21.4–23.8]22.53%yes
1.462.7% [61.3–64.0]62.55%yes

Every rate above is derived from these per-seed counts.

Per-seed counts and the spread across seeds

Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.

Per-seed counts for every cell and outcome
celloutcomeseedsper-seed countssd / binomial
lrt-0.7reject5050…5054737 · 740 · 727 · 752 · 7360.65
lrt-0.8reject5150…5154394 · 371 · 384 · 393 · 3750.67
lrt-0.9reject5250…5254139 · 119 · 131 · 149 · 1311.03
lrt-1reject5350…535452 · 45 · 58 · 44 · 550.88
lrt-1.1reject5450…545492 · 100 · 81 · 104 · 980.96
lrt-1.2reject5550…5554239 · 222 · 224 · 224 · 2200.57
lrt-1.4reject5650…5654629 · 619 · 613 · 624 · 6490.90

Explain this resultoptional · your own key

Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.