Statistics Lab / Appendix
Simulation studies
The 2020 answers applied each method once, to one dataset. These studies apply five of them thousands of times to data whose truth is known, and count how often an interval covers or a test rejects. Every rate comes with a Wilson 95% interval and a Monte Carlo standard error, so the simulations are held to the same standard as the statistics they check.
- 5 studies · 49 cells
- design declared in code
- 245,000 replications
- 5 seed blocks × 1,000 per cell
- 51 of 51 exact values
- inside their Wilson 95% intervals
- Every count replayed in R
- scripts/r_studies.R, identical
How to read these studies
Design, seeds and uncertainty
Each cell is one method under one data-generating setting. It runs five independent blocks of 1,000 replications, and block k of cell c starts from set.seed(base + 100c + k), so the seeds are on the page and anyone can rerun a block in R.
A rate of 95% from 5,000 replications has a Monte Carlo standard error of about 0.3 percentage points, , and a Wilson interval about ±0.6 points wide. Where theory gives the exact rate it is marked ◇, and it should fall inside the interval about 19 times in 20.
When two methods are compared they run on the same simulated datasets. The difference is reported with Newcombe’s paired interval and an exact McNemar test, which uses the pairing instead of comparing two separate intervals.
§B.1Coursework A2 Q2(a)
Does the t interval keep its 95%?
The 2020 interval for a mean walking time, , is exact when the data are normal, because then has a t distribution. Real times are often skewed. This study draws samples from a normal, an exponential and a lognormal distribution and checks how often the interval catches the true mean.
Committed resultscommitted, replayed in R
12 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(20005 + 100c + k).
Replications per block (study B.1)
- Normal(15, 3²)
- Exponential, mean 15
- Lognormal(0, 1)
- Exact (normal data)
- Nominal 95%
| data | n | coverage [Wilson 95%] ◇ exact | interval below μ | interval above μ | MCSE |
|---|---|---|---|---|---|
| Normal(15, 3²) | 5 | 95.2% [94.6–95.8]◇ 95.00% | 2.4% | 2.4% | 0.30% |
| Normal(15, 3²) | 10 | 95.0% [94.4–95.6]◇ 95.00% | 2.2% | 2.8% | 0.31% |
| Normal(15, 3²) | 20 | 95.0% [94.4–95.6]◇ 95.00% | 2.3% | 2.7% | 0.31% |
| Normal(15, 3²) | 50 | 94.6% [93.9–95.2]◇ 95.00% | 2.7% | 2.7% | 0.32% |
| Exponential, mean 15 | 5 | 89.1% [88.2–89.9] | 10.4% | 0.5% | 0.44% |
| Exponential, mean 15 | 10 | 90.2% [89.4–91.0] | 9.3% | 0.5% | 0.42% |
| Exponential, mean 15 | 20 | 91.8% [91.0–92.5] | 7.8% | 0.5% | 0.39% |
| Exponential, mean 15 | 50 | 93.2% [92.5–93.9] | 5.9% | 0.9% | 0.36% |
| Lognormal(0, 1) | 5 | 81.9% [80.8–82.9] | 18.0% | 0.1% | 0.54% |
| Lognormal(0, 1) | 10 | 84.1% [83.1–85.1] | 15.8% | 0.1% | 0.52% |
| Lognormal(0, 1) | 20 | 87.4% [86.5–88.3] | 12.5% | 0.1% | 0.47% |
| Lognormal(0, 1) | 50 | 89.8% [88.9–90.6] | 9.9% | 0.3% | 0.43% |
Every rate above is derived from these per-seed counts.
Per-seed counts and the spread across seeds
Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.
| cell | outcome | seeds | per-seed counts | sd / binomial |
|---|---|---|---|---|
| t-normal-n5 | cover | 20005…20009 | 960 · 949 · 959 · 941 · 953 | 1.16 |
| t-normal-n5 | below | 20005…20009 | 18 · 29 · 22 · 27 · 24 | 0.89 |
| t-normal-n5 | above | 20005…20009 | 22 · 22 · 19 · 32 · 23 | 1.03 |
| t-normal-n10 | cover | 20105…20109 | 950 · 951 · 963 · 943 · 945 | 1.14 |
| t-normal-n10 | below | 20105…20109 | 23 · 25 · 16 · 24 · 22 | 0.76 |
| t-normal-n10 | above | 20105…20109 | 27 · 24 · 21 · 33 · 33 | 1.04 |
| t-normal-n20 | cover | 20205…20209 | 949 · 940 · 960 · 947 · 955 | 1.11 |
| t-normal-n20 | below | 20205…20209 | 18 · 29 · 22 · 23 · 24 | 0.83 |
| t-normal-n20 | above | 20205…20209 | 33 · 31 · 18 · 30 · 21 | 1.31 |
| t-normal-n50 | cover | 20305…20309 | 952 · 944 · 951 · 941 · 942 | 0.72 |
| t-normal-n50 | below | 20305…20309 | 21 · 28 · 27 · 29 · 30 | 0.69 |
| t-normal-n50 | above | 20305…20309 | 27 · 28 · 22 · 30 · 28 | 0.59 |
| t-exponential-n5 | cover | 20405…20409 | 887 · 892 · 883 · 912 · 880 | 1.29 |
| t-exponential-n5 | below | 20405…20409 | 107 · 102 · 112 · 85 · 114 | 1.20 |
| t-exponential-n5 | above | 20405…20409 | 6 · 6 · 5 · 3 · 6 | 0.57 |
| t-exponential-n10 | cover | 20505…20509 | 897 · 891 · 907 · 906 · 910 | 0.84 |
| t-exponential-n10 | below | 20505…20509 | 96 · 104 · 90 · 86 · 89 | 0.78 |
| t-exponential-n10 | above | 20505…20509 | 7 · 5 · 3 · 8 · 1 | 1.31 |
| t-exponential-n20 | cover | 20605…20609 | 911 · 898 · 919 · 933 · 928 | 1.60 |
| t-exponential-n20 | below | 20605…20609 | 79 · 97 · 77 · 65 · 70 | 1.44 |
| t-exponential-n20 | above | 20605…20609 | 10 · 5 · 4 · 2 · 2 | 1.54 |
| t-exponential-n50 | cover | 20705…20709 | 941 · 949 · 934 · 923 · 914 | 1.76 |
| t-exponential-n50 | below | 20705…20709 | 48 · 47 · 54 · 68 · 76 | 1.73 |
| t-exponential-n50 | above | 20705…20709 | 11 · 4 · 12 · 9 · 10 | 1.03 |
| t-lognormal-n5 | cover | 20805…20809 | 800 · 824 · 823 · 818 · 829 | 0.92 |
| t-lognormal-n5 | below | 20805…20809 | 199 · 176 · 176 · 181 · 170 | 0.91 |
| t-lognormal-n5 | above | 20805…20809 | 1 · 0 · 1 · 1 · 1 | 0.50 |
| t-lognormal-n10 | cover | 20905…20909 | 829 · 837 · 839 · 856 · 846 | 0.88 |
| t-lognormal-n10 | below | 20905…20909 | 171 · 163 · 160 · 142 · 153 | 0.95 |
| t-lognormal-n10 | above | 20905…20909 | 0 · 0 · 1 · 2 · 1 | 0.94 |
| t-lognormal-n20 | cover | 21005…21009 | 878 · 873 · 862 · 886 · 871 | 0.84 |
| t-lognormal-n20 | below | 21005…21009 | 122 · 126 · 137 · 113 · 128 | 0.84 |
| t-lognormal-n20 | above | 21005…21009 | 0 · 1 · 1 · 1 · 1 | 0.50 |
| t-lognormal-n50 | cover | 21105…21109 | 888 · 902 · 898 · 905 · 896 | 0.68 |
| t-lognormal-n50 | below | 21105…21109 | 108 · 95 · 101 · 94 · 99 | 0.59 |
| t-lognormal-n50 | above | 21105…21109 | 4 · 3 · 1 · 1 · 5 | 1.07 |
Explain this resultoptional · your own key
Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.
§B.2Coursework A1 Q2(c)
The lognormal variance interval, as submitted
The 2020 code estimated λ = σ² of log X by its MLE, , and then used the pivot built for the unbiased :
The result is the textbook interval shrunk by (n − 1)/n. Its exact behaviour follows from , and the simulation checks it on samples drawn under the lognormal model.
Committed resultscommitted, replayed in R
3 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(1520 + 100c + k).
Replications per block (study B.2)
- 2020 interval lies below λ
- 2020 interval lies above λ
- Exact, from χ²
- 2.5% each side if equal-tailed
| n | 2020 interval [Wilson] ◇ exact | below / above λ (exact) | textbook SS/χ² [Wilson] ◇ exact | 2020 − textbook, points |
|---|---|---|---|---|
| 5 | 95.5% [94.9–96.0]◇ 95.49% | 3.76% / 0.75% | 95.2% [94.6–95.7]◇ 95.00% | 0.3 (−0.2 to 0.8) |
| 15 | 94.9% [94.2–95.4]◇ 95.14% | 3.43% / 1.43% | 94.9% [94.2–95.4]◇ 95.00% | 0.0 (−0.4 to 0.4) |
| 50 | 94.6% [93.9–95.2]◇ 95.04% | 3.05% / 1.91% | 94.4% [93.8–95.0]◇ 95.00% | 0.1 (−0.2 to 0.4) |
Every rate above is derived from these per-seed counts.
Per-seed counts and the spread across seeds
Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.
| cell | outcome | seeds | per-seed counts | sd / binomial |
|---|---|---|---|---|
| lambda-n5 | submitted | 1520…1524 | 951 · 954 · 968 · 950 · 952 | 1.13 |
| lambda-n5 | submittedBelow | 1520…1524 | 41 · 33 · 25 · 44 · 38 | 1.26 |
| lambda-n5 | submittedAbove | 1520…1524 | 8 · 13 · 7 · 6 · 10 | 0.94 |
| lambda-n5 | textbook | 1520…1524 | 950 · 950 · 965 · 949 · 945 | 1.13 |
| lambda-n5 | textbookBelow | 1520…1524 | 25 · 22 · 15 · 25 · 24 | 0.90 |
| lambda-n5 | textbookAbove | 1520…1524 | 25 · 28 · 20 · 26 · 31 | 0.81 |
| lambda-n15 | submitted | 1620…1624 | 950 · 944 · 942 · 949 · 958 | 0.89 |
| lambda-n15 | submittedBelow | 1620…1624 | 36 · 41 · 45 · 36 · 24 | 1.33 |
| lambda-n15 | submittedAbove | 1620…1624 | 14 · 15 · 13 · 15 · 18 | 0.49 |
| lambda-n15 | textbook | 1620…1624 | 946 · 940 · 947 · 951 · 959 | 1.01 |
| lambda-n15 | textbookBelow | 1620…1624 | 26 · 25 · 33 · 23 · 15 | 1.33 |
| lambda-n15 | textbookAbove | 1620…1624 | 28 · 35 · 20 · 26 · 26 | 1.05 |
| lambda-n50 | submitted | 1720…1724 | 938 · 947 · 958 · 946 · 940 | 1.09 |
| lambda-n50 | submittedBelow | 1720…1724 | 39 · 28 · 18 · 38 · 37 | 1.61 |
| lambda-n50 | submittedAbove | 1720…1724 | 23 · 25 · 24 · 16 · 23 | 0.76 |
| lambda-n50 | textbook | 1720…1724 | 936 · 945 · 952 · 948 · 941 | 0.85 |
| lambda-n50 | textbookBelow | 1720…1724 | 33 · 25 · 16 · 30 · 31 | 1.33 |
| lambda-n50 | textbookAbove | 1720…1724 | 31 · 30 · 32 · 22 · 28 | 0.75 |
Explain this resultoptional · your own key
Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.
§B.3Coursework A2 Q1
Wald or Agresti–Caffo for two proportions?
The council survey compared 520 of 800 with 600 of 1,000 using the Wald interval . The Agresti–Caffo interval adds one success and one failure to each group before using the same formula. Exact coverage comes from summing binomial probabilities over every possible pair of counts, and the simulation runs both intervals on the same surveys.
Committed resultscommitted, replayed in R
3 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(800 + 100c + k).
Replications per block (study B.3)
- Wald (2020)
- Agresti–Caffo
- Exact, by enumeration
- Nominal 95%
| groups | Wald [Wilson] ◇ exact | Agresti–Caffo [Wilson] ◇ exact | AC − Wald, points (Newcombe) | surveys where they disagree |
|---|---|---|---|---|
| 16, 20 | 94.0% [93.3–94.6]◇ 93.83% | 95.6% [95.0–96.1]◇ 95.58% | 1.6 (1.3 to 2.0) | 80 |
| 40, 50 | 94.5% [93.9–95.1]◇ 94.42% | 95.2% [94.6–95.8]◇ 95.13% | 0.7 (0.4 to 0.9) | 33 |
| 800, 1000 | 94.8% [94.2–95.4]◇ 94.98% | 94.8% [94.2–95.4]◇ 94.99% | 0.0 (−0.1 to 0.1) | 2 |
Every rate above is derived from these per-seed counts.
Per-seed counts and the spread across seeds
Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.
| cell | outcome | seeds | per-seed counts | sd / binomial |
|---|---|---|---|---|
| prop-16-20 | wald | 800…804 | 947 · 940 · 927 · 939 · 946 | 1.06 |
| prop-16-20 | ac | 800…804 | 959 · 958 · 941 · 963 · 958 | 1.31 |
| prop-40-50 | wald | 900…904 | 948 · 939 · 955 · 943 · 942 | 0.87 |
| prop-40-50 | ac | 900…904 | 951 · 948 · 959 · 951 · 951 | 0.61 |
| prop-800-1000 | wald | 1000…1004 | 944 · 938 · 952 · 958 · 949 | 1.09 |
| prop-800-1000 | ac | 1000…1004 | 944 · 938 · 951 · 959 · 949 | 1.12 |
Explain this resultoptional · your own key
Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.
§B.4Coursework A2 Q2(c), A3 Q1(b)
Pooled t or Wilcoxon?
The coursework compared two groups twice, once with a pooled t interval and once with a Wilcoxon rank-sum test on two weeks of 7 days. This study runs both tests at α = 5% on the same datasets, for normal, heavy-tailed and skewed data, shifting the second group by 0 to 1.5 standard deviations.
With 7 per group the exact Wilcoxon test cannot reject at exactly 5%. Its largest rejection region below 5% has probability 3.79%, its attainable size.
Committed resultscommitted, replayed in R
24 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(3000 + 100c + k).
Replications per block (study B.4)
Observations per group
- Pooled t test
- Wilcoxon rank-sum (exact)
- α = 5%
Normal
t, 3 df (heavy tails)
Exponential (skewed)
| data | shift | pooled t [Wilson] | Wilcoxon [Wilson] ◇ exact size | W − t, points (Newcombe) | McNemar p |
|---|---|---|---|---|---|
| Normal | 0 | 5.1% [4.5–5.7] | 3.8% [3.3–4.3]◇ 3.79% | −1.3 (−1.7 to −0.9) | < 0.001 |
| Normal | 0.5 | 13.8% [12.8–14.7] | 10.7% [9.9–11.6] | −3.0 (−3.6 to −2.4) | < 0.001 |
| Normal | 1 | 40.8% [39.5–42.2] | 34.2% [32.9–35.5] | −6.7 (−7.5 to −5.8) | < 0.001 |
| Normal | 1.5 | 73.5% [72.2–74.7] | 66.3% [65.0–67.6] | −7.2 (−8.0 to −6.4) | < 0.001 |
| t, 3 df (heavy tails) | 0 | 4.0% [3.5–4.6] | 3.7% [3.2–4.3]◇ 3.79% | −0.3 (−0.7 to 0.1) | 0.149 |
| t, 3 df (heavy tails) | 0.5 | 19.8% [18.7–20.9] | 18.1% [17.1–19.2] | −1.7 (−2.4 to −1.0) | < 0.001 |
| t, 3 df (heavy tails) | 1 | 55.4% [54.0–56.8] | 55.1% [53.7–56.5] | −0.3 (−1.2 to 0.6) | 0.518 |
| t, 3 df (heavy tails) | 1.5 | 82.6% [81.5–83.6] | 82.6% [81.5–83.6] | 0.0 (−0.8 to 0.8) | 0.961 |
| Exponential (skewed) | 0 | 3.7% [3.2–4.2] | 3.4% [2.9–3.9]◇ 3.79% | −0.3 (−0.8 to 0.1) | 0.167 |
| Exponential (skewed) | 0.5 | 17.0% [16.0–18.0] | 19.9% [18.8–21.0] | 2.9 (2.1 to 3.7) | < 0.001 |
| Exponential (skewed) | 1 | 49.2% [47.8–50.6] | 50.7% [49.3–52.1] | 1.5 (0.6 to 2.5) | 0.002 |
| Exponential (skewed) | 1.5 | 77.3% [76.1–78.5] | 77.1% [76.0–78.3] | −0.2 (−1.1 to 0.7) | 0.732 |
Every rate above is derived from these per-seed counts.
Per-seed counts and the spread across seeds
Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.
| cell | outcome | seeds | per-seed counts | sd / binomial |
|---|---|---|---|---|
| power-normal-n7-d0 | t | 3000…3004 | 55 · 50 · 48 · 49 · 52 | 0.40 |
| power-normal-n7-d0 | wilcoxon | 3000…3004 | 45 · 33 · 41 · 32 · 38 | 0.90 |
| power-normal-n7-d0.5 | t | 3100…3104 | 140 · 136 · 130 · 136 · 146 | 0.54 |
| power-normal-n7-d0.5 | wilcoxon | 3100…3104 | 111 · 109 · 105 · 107 · 105 | 0.27 |
| power-normal-n7-d1 | t | 3200…3204 | 410 · 409 · 433 · 400 · 389 | 1.04 |
| power-normal-n7-d1 | wilcoxon | 3200…3204 | 353 · 347 · 357 · 334 · 317 | 1.09 |
| power-normal-n7-d1.5 | t | 3300…3304 | 731 · 750 · 731 · 722 · 739 | 0.75 |
| power-normal-n7-d1.5 | wilcoxon | 3300…3304 | 653 · 674 · 654 · 651 · 682 | 0.95 |
| power-t3-n7-d0 | t | 3400…3404 | 40 · 39 · 42 · 41 · 40 | 0.18 |
| power-t3-n7-d0 | wilcoxon | 3400…3404 | 36 · 41 · 34 · 41 · 34 | 0.60 |
| power-t3-n7-d0.5 | t | 3500…3504 | 194 · 195 · 204 · 182 · 216 | 1.00 |
| power-t3-n7-d0.5 | wilcoxon | 3500…3504 | 187 · 170 · 184 · 169 · 197 | 0.98 |
| power-t3-n7-d1 | t | 3600…3604 | 537 · 536 · 594 · 574 · 529 | 1.81 |
| power-t3-n7-d1 | wilcoxon | 3600…3604 | 527 · 546 · 595 · 568 · 518 | 1.99 |
| power-t3-n7-d1.5 | t | 3700…3704 | 838 · 820 · 809 · 829 · 832 | 0.94 |
| power-t3-n7-d1.5 | wilcoxon | 3700…3704 | 835 · 818 · 818 · 839 · 820 | 0.85 |
| power-exponential-n7-d0 | t | 3800…3804 | 41 · 29 · 36 · 43 · 35 | 0.92 |
| power-exponential-n7-d0 | wilcoxon | 3800…3804 | 36 · 27 · 33 · 38 · 34 | 0.73 |
| power-exponential-n7-d0.5 | t | 3900…3904 | 170 · 154 · 177 · 177 · 171 | 0.79 |
| power-exponential-n7-d0.5 | wilcoxon | 3900…3904 | 198 · 191 · 200 · 206 · 200 | 0.43 |
| power-exponential-n7-d1 | t | 4000…4004 | 497 · 506 · 487 · 509 · 460 | 1.25 |
| power-exponential-n7-d1 | wilcoxon | 4000…4004 | 510 · 532 · 499 · 522 · 473 | 1.44 |
| power-exponential-n7-d1.5 | t | 4100…4104 | 779 · 771 · 756 · 780 · 780 | 0.78 |
| power-exponential-n7-d1.5 | wilcoxon | 4100…4104 | 767 · 782 · 759 · 779 · 770 | 0.70 |
| power-normal-n20-d0 | t | 4200…4204 | 48 · 57 · 61 · 38 · 45 | 1.35 |
| power-normal-n20-d0 | wilcoxon | 4200…4204 | 43 · 48 · 59 · 46 · 45 | 0.93 |
| power-normal-n20-d0.5 | t | 4300…4304 | 352 · 344 · 332 · 353 · 352 | 0.59 |
| power-normal-n20-d0.5 | wilcoxon | 4300…4304 | 336 · 331 · 319 · 329 · 332 | 0.43 |
| power-normal-n20-d1 | t | 4400…4404 | 870 · 860 · 869 · 881 · 888 | 1.04 |
| power-normal-n20-d1 | wilcoxon | 4400…4404 | 859 · 847 · 852 · 860 · 863 | 0.59 |
| power-normal-n20-d1.5 | t | 4500…4504 | 999 · 998 · 998 · 994 · 995 | 1.21 |
| power-normal-n20-d1.5 | wilcoxon | 4500…4504 | 993 · 997 · 999 · 993 · 993 | 1.27 |
| power-t3-n20-d0 | t | 4600…4604 | 49 · 43 · 34 · 63 · 56 | 1.65 |
| power-t3-n20-d0 | wilcoxon | 4600…4604 | 55 · 42 · 47 · 67 · 61 | 1.41 |
| power-t3-n20-d0.5 | t | 4700…4704 | 422 · 442 · 464 · 441 · 419 | 1.16 |
| power-t3-n20-d0.5 | wilcoxon | 4700…4704 | 562 · 571 · 537 · 542 · 555 | 0.89 |
| power-t3-n20-d1 | t | 4800…4804 | 881 · 887 · 874 · 894 · 891 | 0.80 |
| power-t3-n20-d1 | wilcoxon | 4800…4804 | 968 · 965 · 979 · 972 · 969 | 1.00 |
| power-t3-n20-d1.5 | t | 4900…4904 | 982 · 988 · 987 · 975 · 976 | 1.42 |
| power-t3-n20-d1.5 | wilcoxon | 4900…4904 | 1000 · 1000 · 1000 · 998 · 999 | 1.16 |
| power-exponential-n20-d0 | t | 5000…5004 | 44 · 42 · 36 · 56 · 32 | 1.44 |
| power-exponential-n20-d0 | wilcoxon | 5000…5004 | 54 · 40 · 38 · 53 · 46 | 1.10 |
| power-exponential-n20-d0.5 | t | 5100…5104 | 371 · 388 · 361 · 343 · 380 | 1.15 |
| power-exponential-n20-d0.5 | wilcoxon | 5100…5104 | 576 · 594 · 595 · 564 · 599 | 0.96 |
| power-exponential-n20-d1 | t | 5200…5204 | 883 · 876 · 861 · 865 · 856 | 1.04 |
| power-exponential-n20-d1 | wilcoxon | 5200…5204 | 966 · 968 · 951 · 956 · 959 | 1.14 |
| power-exponential-n20-d1.5 | t | 5300…5304 | 981 · 987 · 985 · 988 · 991 | 1.01 |
| power-exponential-n20-d1.5 | wilcoxon | 5300…5304 | 996 · 999 · 1000 · 998 · 998 | 1.11 |
Explain this resultoptional · your own key
Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.
§B.5Coursework A3 Q5
Checking the power curve of the rate test
For an exponential sample of n = 50, the 2020 answer tested λ = 1 with an equal-tailed version of the likelihood-ratio test: reject when falls outside the 2.5% and 97.5% points of Gamma(50, 1). Its power at any λ then follows from Gamma(50, λ). Simulating the test directly is an independent check of that algebra.
Committed resultscommitted, replayed in R
7 cells × 5 seed blocks × 1,000 replications. Block k of cell c uses set.seed(5050 + 100c + k).
Replications per block (study B.5)
- Analytic power from Gamma(50, λ)
- Simulated, 5,000 samples each
- α = 5%
| true λ | simulated [Wilson] | exact | inside interval |
|---|---|---|---|
| 0.7 | 73.8% [72.6–75.0] | 73.64% | yes |
| 0.8 | 38.3% [37.0–39.7] | 38.18% | yes |
| 0.9 | 13.4% [12.5–14.4] | 12.71% | yes |
| 1 | 5.1% [4.5–5.7] | 5.00% | yes |
| 1.1 | 9.5% [8.7–10.3] | 9.36% | yes |
| 1.2 | 22.6% [21.4–23.8] | 22.53% | yes |
| 1.4 | 62.7% [61.3–64.0] | 62.55% | yes |
Every rate above is derived from these per-seed counts.
Per-seed counts and the spread across seeds
Counts out of each block’s replications. “sd / binomial” compares the spread of the five block rates with the Monte Carlo error a binomial model predicts. Values near 1 mean the blocks vary as much as chance alone would make them.
| cell | outcome | seeds | per-seed counts | sd / binomial |
|---|---|---|---|---|
| lrt-0.7 | reject | 5050…5054 | 737 · 740 · 727 · 752 · 736 | 0.65 |
| lrt-0.8 | reject | 5150…5154 | 394 · 371 · 384 · 393 · 375 | 0.67 |
| lrt-0.9 | reject | 5250…5254 | 139 · 119 · 131 · 149 · 131 | 1.03 |
| lrt-1 | reject | 5350…5354 | 52 · 45 · 58 · 44 · 55 | 0.88 |
| lrt-1.1 | reject | 5450…5454 | 92 · 100 · 81 · 104 · 98 | 0.96 |
| lrt-1.2 | reject | 5550…5554 | 239 · 222 · 224 · 224 · 220 | 0.57 |
| lrt-1.4 | reject | 5650…5654 | 629 · 619 · 613 · 624 · 649 | 0.90 |
Explain this resultoptional · your own key
Add your own Anthropic or OpenAI key in AI settings to enable. Nothing is sent without one.