docs/decisions/
Decision records
Each record follows the same format: context, the decision, the options considered, why, what happened and what I would change. A past record is never edited to change its decision. A new record supersedes it.
DR-001accepted2026-10-09
Keep the paired midrange-vs-mean design and report relative efficiency with a bootstrap interval
Decision: the 2020 comparison of the sample midrange and the sample mean (A1 Q4) stays exactly as submitted, including its paired design and its printed variances. Beside it the lab now reports the relative efficiency var(midrange) / var(mean) with a paired bootstrap percentile interval, and the sample mean remains the recommended estimator for normal data.
Scope: /estimators#midrange, web/src/lib/stats/bootstrap.ts
Context
Assignment 1 Q4 asked whether the midrange (the average of the smallest and largest values) or the sample mean is the better estimator of a normal mean for samples of 20. The notebook drew 100 samples with set.seed(2529) and printed a variance of 0.158578 for the midrange and 0.069188 for the mean. It drew a second matrix for the mean but then applied the mean to the first matrix, so both estimators saw the same 100 samples. The 2020 answer reported two variances and no uncertainty, and 100 replications is a small simulation.
Decision
Keep the 2020 numbers and the paired design. Report one effect size, the relative efficiency var(midrange) / var(mean), with a 95% percentile interval from a paired bootstrap that resamples the 100 (midrange, mean) pairs together (B = 2,000, seed 1). State the theoretical variance of the mean beside its simulated value.
Options considered
- Leave the two variances as printed. Faithful, but it says nothing about how far 100 replications can be trusted.
- Treat the two columns as independent and use an F test. This ignores the pairing, which is the strongest part of the design.
- Paired bootstrap of the variance ratio (chosen). Keeps the pairing and needs no normality of the estimates.
- Re-run with many more replications and replace the 2020 numbers. More precise, but it would silently change a submitted result.
Why
Because both estimators were computed on the same samples, the noise that is common to both cancels in their ratio, and resampling pairs keeps that benefit. A percentile interval is honest about the skewed sampling distribution of a variance ratio. The bootstrap uses the site’s port of R’s sample(), so the same loop in R gives the same interval to the last digit, and the test suite checks this.
What happened
- The relative efficiency on the 2020 draws is 2.29, with a 95% bootstrap interval from 1.76 to 3.05. R reproduces both endpoints exactly.
- A reference run of 100,000 samples in R (
set.seed(1), n = 20) gives variances of 0.1425 and 0.0502, a relative efficiency of 2.84. That value sits inside the interval but well above the 2020 point estimate.
- The 2020 variance of the mean, 0.0692, came out 38% above its theoretical value σ²/n = 0.05. With 100 replications the variance of a variance is large, so the 2020 run understated how much better the mean is. The conclusion that the mean is better stands. The size of the advantage was uncertain, and the interval now says so.
What I’d change
- Choose the number of replications from a target Monte Carlo standard error before running, as the simulation studies on
/studies now do.
- Report the theoretical variance σ²/n next to every simulated one by default, so a run that drifts from theory is visible at once.
DR-002accepted2026-10-09
Estimate a uniform endpoint with the scaled maximum, never the method of moments
Decision: for X₁, …, Xₙ from Unif(0, θ) the lab recommends the MSE-optimal scaled maximum (n + 2)/(n + 1) · max(Xᵢ), or the unbiased (n + 1)/n · max(Xᵢ) when unbiasedness is required, and never the method-of-moments estimator 2X̄. Every simulated mean squared error is now shown with its Monte Carlo standard error.
Scope: /estimators#uniform, web/src/lib/stats/bootstrap.ts (mseWithMcse)
Context
Assignment 1 Q3 derived the method-of-moments estimator 2X̄ and the maximum likelihood estimator max(Xᵢ) for a uniform endpoint, their mean squared errors θ²/(3n) and 2θ²/{(n + 1)(n + 2)}, and the multiple of the MLE that minimises MSE, a = (n + 2)/(n + 1). The 2020 answer was a derivation on paper. The 2026 lab adds a simulation of all three estimators, which needs uncertainty attached before it can be compared with the theory.
Decision
Recommend the scaled maximum. Show the simulated MSE of each estimator with its Monte Carlo standard error, sd((θ̂ − θ)²)/√R, next to the theoretical MSE, so a reader can judge whether a gap between the two is noise.
Options considered
- Method of moments, 2X̄. Unbiased and simple, but it ignores that no observation can exceed θ.
- Maximum likelihood, max(Xᵢ). Always below θ, so biased downwards.
- Unbiased MLE, (n + 1)/n · max(Xᵢ). Removes the bias.
- MSE-optimal scaled MLE, (n + 2)/(n + 1) · max(Xᵢ) (chosen as the default). Accepts a little bias for a lower MSE.
Why
The maximum is a sufficient statistic, so any estimator built on X̄ throws information away. Its variance shrinks like 1/n² rather than 1/n. Between the two rescalings the difference is small and the choice depends on the loss. Mean squared error was the criterion in the question, so the MSE-optimal multiple is the default.
What happened
With the lab’s default settings (n = 10, θ = 3, 2,000 samples from set.seed(20005)) the simulated MSEs are 0.288 ± 0.009 for 2X̄, 0.127 ± 0.005 for the maximum and 0.069 ± 0.003 for the scaled maximum, against theoretical values of 0.300, 0.136 and 0.074. The MSE of 2X̄ is 4.19 times that of the scaled maximum, with a paired bootstrap interval from 3.80 to 4.61, and the theoretical ratio of 4.03 lies inside it.
All three simulated MSEs sit between 1.3 and 1.8 standard errors below their theoretical values. They come from the same 2,000 samples, so their errors move together, and a run of slightly small maxima pulls all three down at once. The ranking of the estimators does not depend on it.
What I’d change
- Add the unbiased (n + 1)/n · max(Xᵢ) to the lab as a fourth estimator so the bias-variance trade-off between the two rescalings is visible.
- Use a paired interval for every between-estimator comparison in the lab, not only in this record.
DR-003accepted2026-10-09
Keep the 2020 interval for the lognormal variance and show its unequal tails
Decision: the A1 Q2 interval for λ = σ² stays as submitted, (n − 1)λ̂ / χ², with its printed values (1.118327, 5.189367). It is labelled “as submitted” and shown beside the textbook equal-tailed interval SS / χ². The simulation study on /studies reports the exact and simulated coverage of both, tail by tail.
Scope: /fitting#lognormal, /studies#lambda-interval, web/src/lib/studies/definitions.ts
Context
For a lognormal sample, λ is the variance of log X. The 2020 code computed the MLE λ̂ = SS/n, where SS = Σ(log xᵢ − mean)², and then plugged it into the pivot (n − 1)S²/λ ~ χ²ₙ₋₁, which is built for the unbiased S² = SS/(n − 1). The result is the textbook interval multiplied by (n − 1)/n. Nothing in the 2020 answer was wrong arithmetically, but the interval is not the one the pivot gives.
Decision
Do not change the submitted interval. Add the textbook interval beside it in the lab, and measure both under repeated sampling with exact coverage from the χ² distribution and Monte Carlo coverage from 5,000 samples per sample size.
Options considered
- Replace the 2020 interval with the textbook one. Correct, but it rewrites a submitted result.
- Keep it and say nothing. Faithful, but hides a known flaw.
- Keep it, label it, and measure it (chosen).
Why
The shift by (n − 1)/n moves both cut-offs on SS/λ upwards by n/(n − 1). Whether that matters is an empirical question about coverage, and the χ² distribution answers it exactly, so the honest response is to compute it and show it rather than to assume.
What happened
- Overall coverage of the 2020 interval is close to nominal: 95.49% at n = 5, 95.14% at n = 15 (the coursework’s sample size) and 95.04% at n = 50, from the χ² distribution.
- The tails are not balanced. At n = 15 the 2020 interval lies entirely below λ 3.43% of the time and entirely above it 1.43% of the time, against 2.5% each for the textbook interval. It is shifted downwards, as a plug-in of the downward-biased MLE would suggest.
- The simulation agrees with the exact values in every cell. At n = 15 the 2020 interval covered in 94.86% of 5,000 samples (Wilson 95% interval 94.21% to 95.44%). On the same samples the paired difference in coverage between the two intervals is 0.0 percentage points (−0.4 to 0.4).
- On the 2020 data the textbook interval is (1.1982, 5.5600), against the submitted (1.1183, 5.1894).
- A larger problem showed up in the assumption checks. Shapiro–Wilk on the logs of the 2020 sample gives p = 0.009, so the normal model behind any χ² interval for these data is doubtful (DR-007). That matters more than the divisor.
What I’d change
- Report a one-sided miss rate for every interval in the lab, because an interval can hit 95% overall while missing on one side three times as often as on the other.
- For the 2020 data, a bootstrap interval for the variance of the logs would not lean on normality.
DR-004accepted2026-10-09
Keep the Wald interval for the council survey and recommend Agresti–Caffo for small groups
Decision: the 2020 Wald interval for p₁ − p₂ (A2 Q1, 520 of 800 against 600 of 1,000) stays as submitted, because at those sample sizes it behaves the same as the Agresti–Caffo interval. For groups of a few dozen or fewer the lab recommends Agresti–Caffo, which adds one success and one failure to each group.
Scope: /intervals#proportions, /studies#two-proportion, web/src/lib/stats/proportion.ts
Context
The 2020 answer used prop.test(..., correct = FALSE), whose interval for the difference is the Wald interval p̂₁ − p̂₂ ± z · se. The Wald interval is known to undercover when samples are small or proportions are near 0 or 1. The question is whether that applies at the coursework’s sample sizes.
Decision
Measure both intervals at the coursework’s sizes and at two smaller sizes with the same 4 : 5 ratio, with true proportions p₁ = 0.65 and p₂ = 0.60 (the 2020 estimates). Use exact coverage, from summing binomial probabilities over every possible pair of counts, and Monte Carlo coverage from 5,000 simulated surveys per size, with both intervals computed on the same surveys.
Options considered
- Wald (as submitted).
- Agresti–Caffo (chosen for small groups). One line of code, close to nominal coverage.
- Newcombe’s hybrid score interval. Good coverage, more involved to explain.
- Exact (conditional) intervals. Conservative, and awkward for a difference of two independent proportions.
Why
Agresti–Caffo fixes the Wald interval’s worst behaviour with almost no extra complexity, and it agrees with the Wald interval when samples are large. A paired comparison on the same simulated surveys shows directly how often the choice changes the answer.
What happened
| Groups (n₁, n₂) | Wald, exact | Agresti–Caffo, exact | Agresti–Caffo − Wald on the same surveys |
|---|
| 16, 20 | 93.83% | 95.58% | +1.6 points (1.3 to 2.0) |
| 40, 50 | 94.42% | 95.13% | +0.7 points (0.4 to 0.9) |
| 800, 1000 | 94.98% | 94.99% | 0.0 points (−0.1 to 0.1) |
At the coursework’s sizes the two intervals disagreed about covering the true difference in 2 of 5,000 simulated surveys. The 2020 interval needs no change. At 16 and 20 the Wald interval misses 6.2% of the time instead of 5%.
This study looks at one pair of true proportions only. The Wald interval is worst when a proportion is near 0 or 1, and this design does not probe that region, so it understates the case for Agresti–Caffo in general.
What I’d change
- Extend the grid to proportions near 0.05 and 0.95, where the difference between the intervals is largest.
- Show the Agresti–Caffo interval as a second line in the two-proportion lab.
DR-005accepted2026-10-09
Prefer the Wilcoxon test at about 20 per group for heavy-tailed or skewed data, report both at 7 per group, and state the attainable size
Decision: with about 20 observations per group and data that may be heavy-tailed or skewed, the lab treats the exact Wilcoxon rank-sum test as the default (+11.6 and +21.7 points of power at a shift of 0.5 SD, against −1.7 under normality) and shows the pooled t test beside it. At 7 per group neither test dominates, so the lab reports both. Every exact Wilcoxon result states its attainable size, which is 3.79% rather than 5% for two groups of 7.
Scope: /tests#wilcoxon, /studies#two-sample-power
Context
The coursework used both tests. A2 Q2(c) built a pooled t interval for two walking-time samples, and A3 Q1(b) ran a Wilcoxon rank-sum test on two weeks of daily case counts, 7 days each. Neither answer said why one test was chosen over the other, or what each costs in power.
Decision
Choose between the tests by sample size and by what is known about the shape of the data, using a paired simulation:
- About 20 per group, tails or skew likely: the exact Wilcoxon test is the default, with the pooled t test beside it.
- 7 per group: report both tests and the Wilcoxon test’s attainable size of 3.79%. Neither dominates.
The evidence comes from running both tests on the same simulated datasets, for 7 and 20 observations per group, for normal, heavy-tailed (t with 3 degrees of freedom) and skewed (exponential) data, at shifts of 0, 0.5, 1 and 1.5 standard deviations. Report each test’s rejection rate with a Wilson interval and the paired difference with Newcombe’s interval and an exact McNemar p-value.
Options considered
- Pooled t test. Exact size under normality and equal variances.
- Welch t test. Drops the equal-variance assumption.
- Exact Wilcoxon rank-sum test (chosen as the default at about 20 per group when tails or skew are likely). Distribution-free size.
- Permutation test on the mean difference. Exact size, more computation, harder to explain on paper.
Why
The Wilcoxon test’s size does not depend on the shape of the data, and the study shows its power loss under normality is small at 20 per group while its gain under heavy tails and skew is large. At 7 per group the evidence does not support a default either way. The pooled t test kept Type I error control for all three shapes, and the Wilcoxon test gained power only for skewed data at small shifts. With 7 per group the exact test cannot reach 5%, which costs power under normality, so stating the attainable size is part of reporting it honestly.
What happened
Rejection rates in percentage points, 5,000 datasets per cell:
| Data, n per group | Shift | Pooled t | Wilcoxon | Wilcoxon − t (95% interval) |
|---|
| Normal, 7 | 1 SD | 40.8 | 34.2 | −6.7 (−7.5 to −5.8) |
| Exponential, 7 | 0.5 SD | 17.0 | 19.9 | +2.9 (2.1 to 3.7) |
| t₃, 7 | 1 SD | 55.4 | 55.1 | −0.3 (−1.2 to 0.6) |
| Normal, 20 | 0.5 SD | 34.7 | 32.9 | −1.7 (−2.4 to −1.0) |
| t₃, 20 | 0.5 SD | 43.8 | 55.3 | +11.6 (10.5 to 12.6) |
| Exponential, 20 | 0.5 SD | 36.9 | 58.6 | +21.7 (20.5 to 22.9) |
- Under no shift the Wilcoxon test rejected 3.78% of normal datasets of 7 per group, against an exact attainable size of 3.79%. The t test rejected 5.08%.
- At 7 per group the paired difference clearly favoured the t test in 4 of the 9 shifted cells (normal data at all three shifts, −3.0 to −7.2 points, and t₃ at 0.5 SD, −1.7) and clearly favoured the Wilcoxon test in 2 (exponential data at 0.5 and 1 SD, +2.9 and +1.5). In the other 3 the difference was under half a point and its interval included zero. “Clearly” means Newcombe’s interval excludes zero.
- With 7 per group the t test became conservative on skewed data (3.68% rejections under no shift, Wilson interval 3.19% to 4.24%). It still controls Type I error there, but it loses its exact size, and with it some power.
- At 7 per group the Wilcoxon’s lower attainable size accounts for part of its deficit under normality. A randomised or mid-p version would recover some of it, at the cost of a less familiar test.
What I’d change
- Add the Welch t test. The 2020 walking-time samples had clearly unequal variances (DR-007), a situation this study does not cover because it shifts location only.
- Add scale differences between the groups, where the pooled t test’s size breaks down.
DR-006accepted2026-10-09
Fix every simulation design in code, report Monte Carlo error, and replay it in R
Decision: every simulation study on /studies has its design declared in code: the grid of settings, 5 seed blocks of 1,000 replications per cell, and seeds of the form base + 100 × cell + block. Every rate is shown with a Wilson 95% interval and a Monte Carlo standard error, the spread across the five seeds is shown next to what binomial noise predicts, and an independent R script must reproduce every count exactly. The design and its results were committed together, so this is a fixed, checkable design, not a pre-registration.
Scope: web/src/lib/studies, web/src/content/study-results.json, scripts/r_studies.R
Context
The 2020 coursework ran two small simulations (100 replications each) and printed point estimates without any measure of their own noise. The revival adds five larger studies of coverage, size and power. A simulation result is itself an estimate, and it can mislead in the same ways as any other estimate if its error is not reported or if the design is chosen after looking at the output.
Decision
- Each cell runs 5,000 replications in 5 independent blocks, so the Monte Carlo standard error of a rate near 95% is about 0.3 percentage points and a Wilson interval is about ±0.6 points wide.
- The committed results file holds only raw per-seed counts. Rates, intervals and comparisons are derived from those counts by tested helpers, so nothing on the page is typed in by hand.
- Paired designs are used wherever two procedures are compared, and the comparison uses Newcombe’s paired interval and an exact McNemar test rather than the overlap of two separate intervals.
- Where theory gives an exact rate, it is shown beside the simulated one.
scripts/r_studies.R re-runs every cell in R with R’s own t.test(), wilcox.test(), qchisq() and qgamma(), and the test suite requires identical counts.
Options considered
- A single seed per cell. Simpler, but gives no direct look at seed-to-seed variation.
- Many seeds with few replications each. Shows the spread well but makes each block’s rate very noisy.
- Five blocks of 1,000 (chosen). Enough replications for a ±0.6-point interval, and enough blocks to compare the observed spread with the binomial prediction.
Why
A design declared in code and replayed in R can be checked: anyone can confirm that the published counts come from that grid, those seeds and that number of replications, and from nothing else. It does not prove the design was chosen before any output was seen, because the design and the results arrived in the same commit. Five blocks make the claim “the Monte Carlo error is what the binomial model says” checkable instead of assumed. An independent replay in R is the strongest check available that the simulation does what its description says. It catches a wrong formula, a wrong seed or a wrong draw order, and the R-compatible random number generator makes the replay exact rather than approximate.
What happened
- 49 cells and 245,000 replications in total. R reproduces every per-seed count in every cell. R takes about 70 seconds and the TypeScript engine about 3 seconds.
- 51 rates have an exact value from theory (the t interval under normality, both λ intervals, both two-proportion intervals, the Wilcoxon test’s attainable size and the likelihood-ratio test’s power curve). All 51 exact values lie inside their Wilson intervals. With 95% intervals about 2 or 3 misses would be expected if the rates were independent. Many of them share replications, which makes a clean sweep less surprising than it looks, so I read this as “no sign of miscalibration” rather than as better-than-expected accuracy.
- Across the 115 rates whose spread can be compared, the standard deviation over the five seeds averages 1.04 times the binomial prediction (root mean square of the ratio) and never exceeds 2 times it. With only five blocks each of those ratios is itself noisy.
- The seeds are visible on the page, and a reader can re-run any study in the browser with another seed base.
What I’d change
- Pre-register future studies properly: commit the design (the definitions and the R script) on its own, before generating any results, and cite that commit’s hash in the decision record.
- Compare the 2020 rate test in B.5 with the exact-size likelihood-ratio region. The 2020 answer used equal tails, (37.11, 64.78) for n = 50, while Λ(a) = Λ(b) with size 5% gives (37.37, 65.20). The study as designed checks the algebra of the 2020 test, not whether equal tails were the right choice.
- Choose the number of replications per study from a target interval width rather than one number for every study. The power study would benefit from more replications near 50% power, where binomial error is largest.
- Add a regression test that runs a reduced version of each study under the R replay in CI, so a change to R itself would be noticed. Today the R side runs only when the fixtures are regenerated.
DR-007accepted2026-10-09
Check every assumption the 2020 methods rely on, and publish the checks beside the unchanged results
Decision: each method in the 2020 coursework gets a fixed check of the assumption it depends on. The checks run on the original inputs inside the test suite, only their aggregates are published (on /diagnostics), and they are read with one fixed rule. The 2020 results are never changed because of them.
Scope: /diagnostics, web/src/lib/assumption-checks.ts, web/src/lib/stats/diagnostics.ts, web/src/lib/stats/series.ts
Context
The 2020 answers checked assumptions by eye, with a QQ plot for the plant heights and the four plot(lm) panels for the coffee regression, and ran one formal check, the F test for equal variances in A2 Q2(e). Assignment 3 Q1 asked students to treat 21 consecutive days of case counts as a random sample, and the question itself called that unrealistic. A reader of the revived site should be able to see which 2020 conclusions rest on firm ground and which do not.
Decision
Twelve checks, one per assumption:
| Coursework | Assumption | Check |
|---|
| A1 Q1 | Heights are exponential | Kolmogorov–Smirnov with the rate estimated, p-value by parametric bootstrap (999 resamples) |
| A1 Q2 | Logs are normal | Shapiro–Wilk |
| A2 Q2 | Each walking-time sample is normal | Shapiro–Wilk |
| A2 Q2 | Equal variances | F test (the 2020 answer’s own) |
| A2 Q3 | Normal, constant-variance, independent errors and no dominant point | Shapiro–Wilk, Breusch–Pagan, Durbin–Watson with a Monte Carlo p-value, Bonferroni outlier test |
| A3 Q1 | Days are a random sample | Runs test, Ljung–Box, Mann–Kendall with a Theil–Sen slope |
One reading rule is applied to every check. A p-value below 0.01 is “strong evidence against”, below 0.05 is “some evidence against”, and anything else is “no evidence against”. The last never means the assumption holds.
Options considered
- Leave the visual checks as they were. Faithful, but the case-count problem stays invisible.
- Replace the 2020 analyses where a check fails. That would rewrite submitted work.
- Publish the checks beside the unchanged results (chosen).
Why
Checks beside the results let a reader weigh each 2020 conclusion without the site deciding for them. Running them in the test suite keeps the university datasets out of the deployed site, the same arrangement as the parity ledger. Every statistic is checked against R, or against SciPy for the Theil–Sen interval.
What happened
Five of the twelve checks found strong evidence against an assumption. Seven found none.
- A1 Q2. Shapiro–Wilk on the logs gives W = 0.830, p = 0.009. The lognormal model behind the 2020 χ² interval is doubtful for this sample.
- A2 Q2. The 2020 F test rejected equal variances (F = 15.28, p = 0.0016), yet part (c) used the pooled interval. The Welch interval, (−7.578, −2.952) on 10.4 degrees of freedom, is close to the pooled (−7.719, −2.811), and both exclude zero, so the 2020 conclusion that route Y is slower survives.
- A3 Q1. The case counts fall steadily over the 21 days. There are 4 runs above and below the median where about 11 are expected (exact p = 0.002), Ljung–Box gives Q = 25.8 on 7 lags (p = 0.0005), and Kendall’s τ against time is −0.69 (p = 0.00002). The Theil–Sen slope is −2.21 cases per day (95% interval −2.90 to −1.64). The sign test’s p = 0.9608 for “the median is below 15” therefore answers a question the data cannot pose. There is no single median for a series that is trending downwards.
- No evidence against the exponential model for the plant heights (D = 0.142, bootstrap p = 0.296 ± 0.014), normality of either walking-time sample, or any of the four regression assumptions (Shapiro–Wilk p = 0.99, Breusch–Pagan p = 0.59, Durbin–Watson p = 0.67 ± 0.01, largest studentised residual Bonferroni p = 0.51). These samples have between 8 and 30 values, so the checks have little power, and “no evidence against” is the most they can say.
- With twelve checks at the 5% level, about 0.6 false alarms would be expected by chance. All five flagged checks have p below 0.01, and three of them describe the same trending series.
What I’d change
- Report power for each check at the sample size used, so “no evidence against” can be read as weak or strong.
- For the case counts, replace the two-sample comparison with a model that has a trend in it, such as a Poisson regression on day, if the question were being set today.
DR-008accepted2026-10-09
Optional “Explain this result” with the visitor’s own key, grounded on the numbers on screen and audit-logged locally
Decision: the only AI feature is an optional “Explain this result” panel. It uses the visitor’s own Anthropic or OpenAI key, calls the provider directly from the browser, sends only the numeric summary already shown on the page, labels every output as AI-generated, flags any number in the output that cannot be traced to that summary, and records every call (without the key) in an audit log in the visitor’s IndexedDB. No AI output ever feeds a computed result.
Scope: web/src/lib/ai, web/src/components/ai, /ai-log, the AI use statement on /methods
Context
The upgrade had to show GenAI fluency with governance on a static site that has no server and no budget for AI. The principles it is informed by (the Australian Digital Transformation Agency’s policy for the responsible use of AI in government, the transparency principles of the EU AI Act and the NIST AI Risk Management Framework) point the same way. People should be told when content is AI-generated, a human should stay in the loop, calls should be recorded, and it should be clear what data goes where. The site makes no claim of compliance with any of them.
Decision
Bring your own key only. Keys live in sessionStorage by default, or in localStorage if the visitor ticks “remember on this device”, and one button forgets them. Anthropic calls go through the official @anthropic-ai/sdk with dangerouslyAllowBrowser, which sends the anthropic-dangerous-direct-browser-access header. The SDK is loaded with a dynamic import on the first request, so pages that never use AI do not download it. OpenAI calls use fetch against the Chat Completions API. Replies are constrained with structured outputs, validated with zod, checked for untraceable numbers, and shown with Accept, Edit and Reject controls whose outcome is written to the audit log. The default model is Claude Haiku 4.5, the cheapest, with Claude Sonnet 5.5 as an option. The OpenAI model id is free text.
Options considered
- No AI at all.
- A server-side proxy with a site-owned key. Needs a budget, rate limiting and abuse handling.
- Bring your own key through a proxy. The visitor’s key would pass through infrastructure they do not control.
- Bring your own key, called directly from the browser (chosen).
- An on-device model. Gigabyte downloads and a capable GPU for a few sentences of text.
Why
Option 4 costs nothing to run, keeps the key away from any server of ours and keeps the audit trail with the person who made the calls. The feature is deliberately narrow. It explains results and never produces them, so a wrong explanation cannot change a number on the page. The facts sent are the same numbers the page shows, which makes the grounding check meaningful: any other number in the reply is either a rounding of a fact or something the model made up. The official SDK is used for Anthropic because its typed errors map cleanly to messages a visitor can act on, and the dynamic import keeps that choice from weighing down the static pages.
What happened
- The adapters, key storage, audit store, CSV and JSON export, grounding check and error mapping are covered by 36 unit tests with
fetch mocked, including the exact headers the SDK sends. They have not been run against the live APIs in CI, by design, because there is no key to use. The request shapes follow the providers’ documentation and are checked only through mocks, which is a real limitation.
- The grounding check catches invented numbers but not wrong reasoning built on correct numbers, so it is shown as a warning, never as a guarantee. It is also loose for whole numbers: one of 10 or more passes if it is within 0.5 of a fact or of a proportion × 100. Of the integers from 0 to 100, 28 trace to the t-coverage facts by chance, and 68 to the power-study facts, which hold many rates. Numbers quoted to one decimal are stricter (58 and 182 of the 1,001 values from 0.0 to 100.0). Whole numbers below 10 must match a fact exactly, and the panel’s pass message says that larger ones are only loosely checked.
- Direct browser access means any script running on the page could read the key. The site loads no third-party scripts, keeps the key in sessionStorage by default and sends a Content-Security-Policy that keeps fetches and resource loads on this site, with
connect-src adding only the two provider APIs. That reduces the ways an injected script could send a key elsewhere, but it cannot stop a script from navigating the page away, so it is a mitigation rather than a guarantee.
- Each call is one attempt. The SDK’s automatic retries are switched off because the visitor pays for every request and should see every failure.
- OpenAI’s 401 message echoes a masked fragment of the key. Error text is redacted before it is shown or logged, and a record that contains the key is refused outright.
What I’d change
- Add an evaluation harness if the feature grows: a fixed set of results with reference readings, scored for agreement with the facts across repeated runs and models, with intervals on the scores.
- Tighten the Content-Security-Policy with nonce-based
script-src instead of 'unsafe-inline'.
- Let visitors import an exported audit log into another browser, and add a retention setting.