Hypothesis tests, size and power
A test is a rule for deciding between two hypotheses, and every rule can be wrong in two ways. The first two sections make those error probabilities visible; the last two apply distribution-free tests to a real series — Victoria’s daily COVID-19 case counts during the second-wave lockdown in September 2020.
§4.1Coursework A2 Q5
One observation, two kinds of error
A single observation from an exponential distribution with mean θ is used to test against , rejecting when . Since ,
For a test of size exactly 0.05, solve : .
Drag k to trade Type I error against Type II error.
Type I error α
0.1353
e^(−k/θ₀) = e^(−2.000)
Type II error β
0.5507
1 − e^(−k/θ₁)
Power 1 − β
0.4493
k for α = 0.05
5.991
−θ₀ ln 0.05
- Density under H₀ (θ = 2)
- Density under H₁ (θ = 5)
- α: H₀ density beyond k
- β: H₁ density below k
- Power P(X > k | θ)
- Value at θ₀ is the size α
Show the numbers as a table
| θ | P(reject) |
|---|---|
| 1 | 0.0183 |
| 2 | 0.1353 |
| 3 | 0.2636 |
| 4 | 0.3679 |
| 5 | 0.4493 |
| 6 | 0.5134 |
| 7 | 0.5647 |
| 8 | 0.6065 |
| 9 | 0.6412 |
| 10 | 0.6703 |
| 11 | 0.6951 |
| 12 | 0.7165 |
§4.2Coursework A3 Q5
A likelihood ratio test for a rate
For the likelihood is with , maximised at . The likelihood ratio for depends on the data only through Y, and under H₀
where a and b are the α/2 and 1 − α/2 quantiles of that gamma distribution. With n = 50 and λ₀ = 1 the 2020 answer found a ≈ 37.11 and b ≈ 64.78.
Try it on a simulated sample
Lower critical value a
37.1110
qgamma(0.025, 50, 1)
Upper critical value b
64.7806
qgamma(0.975, 50, 1)
Observed Y = Σxᵢ
46.381
retain H₀
Power at true λ
0.2253
Λ = 0.8714
- Gamma(50, 1) density of Y under H₀
- Rejection region, total α = 0.05
- Observed Y
- Power P(Y < a or Y > b | λ)
- Size α at λ₀
§4.3Coursework A3 Q1(a)
Was the median below 15 cases a day?
The sign test makes almost no assumptions: under each observation is above the median with probability ½, so the number of + signs is . Testing whether the median daily count was below 15, only small values of Y are evidence against H₀:
Fourteen of the 21 days had at least 15 new cases, so there is no evidence that the median was below 15 — although the counts were clearly falling, which the test (treating days as a random sample) ignores.
Victoria, daily new cases, 7–27 September 2020 (one row per week).
Alternative H₁
Values ≥ m₀ (counted +)
14
of n = 21
Values < m₀
7
Y under H₀
Bin(21, ½)
p-value
0.9608
retain H₀ at 5%
- At or above m₀ (+)
- Below m₀ (−)
- m₀
Show the numbers as a table
| day | value | sign |
|---|---|---|
| 7 Sep | 48 | + |
| 8 Sep | 70 | + |
| 9 Sep | 47 | + |
| 10 Sep | 40 | + |
| 11 Sep | 35 | + |
| 12 Sep | 41 | + |
| 13 Sep | 30 | + |
| 14 Sep | 39 | + |
| 15 Sep | 41 | + |
| 16 Sep | 25 | + |
| 17 Sep | 44 | + |
| 18 Sep | 20 | + |
| 19 Sep | 13 | − |
| 20 Sep | 11 | − |
| 21 Sep | 28 | + |
| 22 Sep | 14 | − |
| 23 Sep | 11 | − |
| 24 Sep | 13 | − |
| 25 Sep | 12 | − |
| 26 Sep | 16 | + |
| 27 Sep | 5 | − |
- Bin(n, ½) probabilities
- As or more extreme (p-value)
§4.4Coursework A3 Q1(b)
Did week 2 have more cases than week 3?
The Wilcoxon rank-sum test compares two groups through the ranks of the pooled sample. R reports, the number of (x, y) pairs with x > y (ties count ½). With ties in the data R switches to a normal approximation with a tie-corrected variance and a continuity correction:
Week 2: 14–20 September 2020
Week 3: 21–27 September 2020
Alternative H₁
p-value
Rank sum of X
66
− nₓ(nₓ+1)/2 = 28
W
38
E(W) = 24.5 under H₀
z (corrected)
1.665
ties present
p-value
0.04798
reject H₀ at 5%
- Group X
- Group Y
Show the numbers as a table
| rank | value | group |
|---|---|---|
| 1 | 5 | Y |
| 2.5 | 11 | X |
| 2.5 | 11 | Y |
| 4 | 12 | Y |
| 5.5 | 13 | X |
| 5.5 | 13 | Y |
| 7 | 14 | Y |
| 8 | 16 | Y |
| 9 | 20 | X |
| 10 | 25 | X |
| 11 | 28 | Y |
| 12 | 39 | X |
| 13 | 41 | X |
| 14 | 44 | X |