Skip to content
Statistics LabMAST20005 · 2020

Hypothesis tests, size and power

A test is a rule for deciding between two hypotheses, and every rule can be wrong in two ways. The first two sections make those error probabilities visible; the last two apply distribution-free tests to a real series — Victoria’s daily COVID-19 case counts during the second-wave lockdown in September 2020.

§4.1Coursework A2 Q5

One observation, two kinds of error

A single observation from an exponential distribution with mean θ is used to test H0:θ=2H_0:\theta=2 against H1:θ=5H_1:\theta=5, rejecting when X>4X>4. Since P(X>k∣θ)=e−k/θP(X>k\mid\theta)=e^{-k/\theta},

α=e−4/2=e−2≈0.135,\alpha=e^{-4/2}=e^{-2}\approx0.135,
β=1−e−4/5≈0.551,\beta=1-e^{-4/5}\approx0.551,
1−β=e−4/5≈0.449.1-\beta=e^{-4/5}\approx0.449.

For a test of size exactly 0.05, solve e−k/2=0.05e^{-k/2}=0.05: k=−2log⁡0.05≈5.991k=-2\log 0.05\approx5.991.

2.0
5.0
4.00

Drag k to trade Type I error against Type II error.

Type I error α

0.1353

e^(−k/θ₀) = e^(−2.000)

Type II error β

0.5507

1 − e^(−k/θ₁)

Power 1 − β

0.4493

k for α = 0.05

5.991

−θ₀ ln 0.05

  • Density under H₀ (θ = 2)
  • Density under H₁ (θ = 5)
  • α: H₀ density beyond k
  • β: H₁ density below k
Figure 4.1.The two error probabilities are areas under different curves on opposite sides of the same cut-off. Moving k right shrinks α and grows β.
  • Power P(X > k | θ)
  • Value at θ₀ is the size α
Figure 4.2.The power function of the test for the current k. It passes through α at θ₀ and through 1 − β at θ₁.
Show the numbers as a table
θP(reject)
10.0183
20.1353
30.2636
40.3679
50.4493
60.5134
70.5647
80.6065
90.6412
100.6703
110.6951
120.7165

§4.2Coursework A3 Q5

A likelihood ratio test for a rate

For X1,…,Xn∼Exp(λ)X_1,\dots,X_n\sim\mathrm{Exp}(\lambda) the likelihood is λne−λY\lambda^n e^{-\lambda Y} with Y=∑XiY=\sum X_i, maximised at λ^=n/Y\hat\lambda=n/Y. The likelihood ratio for H0:λ=λ0H_0:\lambda=\lambda_0 depends on the data only through Y, and under H₀

Y∼Gamma(n,λ0),Y\sim\mathrm{Gamma}(n,\lambda_0),
reject H0 if Y<a or Y>b,\text{reject } H_0 \text{ if } Y<a \text{ or } Y>b,

where a and b are the α/2 and 1 − α/2 quantiles of that gamma distribution. With n = 50 and λ₀ = 1 the 2020 answer found a ≈ 37.11 and b ≈ 64.78.

50
1.0
0.05

Try it on a simulated sample

1.20

Lower critical value a

37.1110

qgamma(0.025, 50, 1)

Upper critical value b

64.7806

qgamma(0.975, 50, 1)

Observed Y = Σxᵢ

46.381

retain H₀

Power at true λ

0.2253

Λ = 0.8714

  • Gamma(50, 1) density of Y under H₀
  • Rejection region, total α = 0.05
  • Observed Y
Figure 4.3.The likelihood ratio Λ = (λ₀Y/n)ⁿ e^(n − λ₀Y) is small when Y is far from n/λ₀ in either direction, so the test rejects in both tails of the Gamma(n, λ₀) null distribution.
  • Power P(Y < a or Y > b | λ)
  • Size α at λ₀
Figure 4.4.Power rises on both sides of λ₀ and rises faster for larger n. The equal-tailed test is not exactly unbiased: its minimum sits slightly off λ₀.

§4.3Coursework A3 Q1(a)

Was the median below 15 cases a day?

The sign test makes almost no assumptions: under H0:m=m0H_0: m = m_0 each observation is above the median with probability ½, so the number of + signs is Y∼Bin(n,12)Y\sim\mathrm{Bin}(n,\tfrac12). Testing whether the median daily count was below 15, only small values of Y are evidence against H₀:

p=P(Y≤14∣p=12)=0.9608.p=P(Y\le 14\mid p=\tfrac12)=0.9608.

Fourteen of the 21 days had at least 15 new cases, so there is no evidence that the median was below 15 — although the counts were clearly falling, which the test (treating days as a random sample) ignores.

Victoria, daily new cases, 7–27 September 2020 (one row per week).

Alternative H₁

Values ≥ m₀ (counted +)

14

of n = 21

Values < m₀

7

Y under H₀

Bin(21, ½)

p-value

0.9608

retain H₀ at 5%

  • At or above m₀ (+)
  • Below m₀ (−)
  • m₀
m₀ = 15
Figure 4.5.The sign test throws away magnitudes and keeps only which side of m₀ each value falls on. Ties with m₀ count as + here, as they did in the 2020 notebook.
Show the numbers as a table
dayvaluesign
7 Sep48+
8 Sep70+
9 Sep47+
10 Sep40+
11 Sep35+
12 Sep41+
13 Sep30+
14 Sep39+
15 Sep41+
16 Sep25+
17 Sep44+
18 Sep20+
19 Sep13−
20 Sep11−
21 Sep28+
22 Sep14−
23 Sep11−
24 Sep13−
25 Sep12−
26 Sep16+
27 Sep5−
  • Bin(n, ½) probabilities
  • As or more extreme (p-value)
Figure 4.6.Under H₀ each value is equally likely to fall either side of the median, so the number of + signs is Bin(n, ½). The p-value adds up the shaded bars.

§4.4Coursework A3 Q1(b)

Did week 2 have more cases than week 3?

The Wilcoxon rank-sum test compares two groups through the ranks of the pooled sample. R reports  W=∑i∈Xrank⁡(xi)−nX(nX+1)2  \;W=\sum_{i\in X}\operatorname{rank}(x_i)-\tfrac{n_X(n_X+1)}{2}\;, the number of (x, y) pairs with x > y (ties count ½). With ties in the data R switches to a normal approximation with a tie-corrected variance and a continuity correction:

z=W−μW−12σW,z=\frac{W-\mu_W-\tfrac12}{\sigma_W},
μW=nXnY2,\mu_W=\tfrac{n_Xn_Y}{2},
σW2=nXnY12(N+1−∑(tj3−tj)N(N−1)),\sigma_W^2=\tfrac{n_Xn_Y}{12}\Big(N+1-\tfrac{\sum (t_j^3-t_j)}{N(N-1)}\Big),
N=nX+nY.N=n_X+n_Y.

Week 2: 14–20 September 2020

Week 3: 21–27 September 2020

Alternative H₁

p-value

Rank sum of X

66

− nₓ(nₓ+1)/2 = 28

W

38

E(W) = 24.5 under H₀

z (corrected)

1.665

ties present

p-value

0.04798

reject H₀ at 5%

  • Group X
  • Group Y
Figure 4.7.All 14 values pooled and ranked from smallest to largest; tied values share the average of their ranks (stacked dots). If X tends to be larger, its dots crowd the right-hand end and W is large. Wilcoxon rank sum test with continuity correction.
Show the numbers as a table
rankvaluegroup
15Y
2.511X
2.511Y
412Y
5.513X
5.513Y
714Y
816Y
920X
1025X
1128Y
1239X
1341X
1444X