Chapter 8

Hypothesis Testing

Confidence intervals, the bootstrap, significance tests, and how many examples a model comparison needs.

A benchmark score is an estimate, not a property of a model. In 2026, LLM systems are often separated by a few percentage points on finite evaluation sets, so sampling noise can decide which line of a leaderboard looks best. Hypothesis testing gives the small set of tools needed to attach uncertainty to those scores and to compare two models on the same examples.

8.1 Sampling distributions and standard error

Treat each evaluation example as a draw from a population of tasks. For accuracy, define $Y_i = 1$ when the model is correct and $0$ otherwise. The measured accuracy is the sample mean $\hat p = \frac1N \sum_i Y_i$ . If the examples are independent and each has success probability $p$ , then

\Var[\hat p] = \frac{p(1-p)}{N}, \qquad \mathrm{SE}(\hat p) \approx \sqrt{\frac{\hat p(1-\hat p)}{N}} .\tag{8.1}

The derivation is only the variance-of-a-mean identity from Section 6.3: Bernoulli variables have variance $p(1-p)$ , and averaging $N$ independent copies divides variance by $N$ . The square root is the standard error, the typical wiggle of $\hat p$ across repeated evaluation sets. The population is an abstraction: it might be all user questions your product will face, all programming issues in a domain, or all prompts matching the benchmark’s sampling recipe. The standard error is therefore conditional on that sampling story; it does not account for benchmark leakage, bad labels, or distribution shift. For 248 correct answers out of 400, $\hat p = 0.62$ and the estimated standard error is 0.024.

Listing 8.1 Standard error and a normal interval for accuracy

def accuracy_standard_error(correct):
    """Standard error of the mean of 0/1 correctness indicators."""
    correct = np.asarray(correct, dtype=np.float64)
    p_hat = correct.mean()
    return np.sqrt(p_hat * (1.0 - p_hat) / correct.size)


def normal_accuracy_ci(correct, z=1.96):
    """Normal-approximation confidence interval for an accuracy."""
    correct = np.asarray(correct, dtype=np.float64)
    p_hat = correct.mean()
    half_width = z * accuracy_standard_error(correct)
    return p_hat - half_width, p_hat + half_width

8.2 Confidence intervals

A confidence interval is a procedure, not a probability statement about this one model. A 95% procedure covers the population score in 95% of repeated evaluation sets, if its assumptions hold. The central limit theorem makes a normal interval reasonable for many benchmark accuracies:

\hat p \pm 1.96\,\widehat{\mathrm{SE}}(\hat p) .\tag{8.2}

For the 248-of-400 example, this interval runs from 0.572 to 0.668. It is wide enough that a reported accuracy of 0.64 from another run is not obviously better.

The bootstrap replaces the normal approximation with resampling. Resample the evaluation rows with replacement, recompute the statistic, and take quantiles of those bootstrap statistics. It works for accuracy, median judge score, pass@k variants, or any metric that is a function of examples, as introduced by Efron [efron1979]. The bootstrap assumes the observed rows are representative enough that drawing from them mimics drawing from the population. It cannot fix a benchmark whose rows all test the wrong skill, but it can expose how much the reported score moves when the finite set is perturbed.

Listing 8.2 A percentile bootstrap interval

def bootstrap_ci(values, statistic=np.mean, draws=2000, level=0.95, rng=None):
    """Percentile bootstrap interval for any statistic of examples."""
    values = np.asarray(values)
    rng = np.random.default_rng(0) if rng is None else rng
    n = len(values)
    stats = np.empty(draws, dtype=np.float64)
    for draw in range(draws):
        sample = values[rng.integers(0, n, size=n)]
        stats[draw] = statistic(sample)
    alpha = (1.0 - level) / 2.0
    return tuple(np.quantile(stats, [alpha, 1.0 - alpha]))

8.3 Significance tests and p-values

A significance test starts with a null hypothesis $H_0$ , chooses a statistic $T$ , and asks how extreme the observed statistic would be if $H_0$ were true:

p\text{-value} = P_{H_0}(|T| \ge |T_{\mathrm{obs}}|) .\tag{8.3}

For two models on the same examples, a permutation test uses the null hypothesis that the two labels are exchangeable within each row. Flip a fair coin for every row, swap the two models' correctness on heads, and recompute the accuracy gap. The p-value is the fraction of shuffled gaps at least as large as the observed one. It is not the probability that the null is true; it is the probability of data this extreme under the null. Small p-values are strongest when the test was chosen before seeing the results; after-the-fact slicing changes the game.

8.4 Paired comparisons

Do not compare two benchmark accuracies as if they came from unrelated test sets when both models answered the same prompts. Most examples are easy or hard for both models, so the paired differences have lower noise than two independent accuracies.

The paired bootstrap resamples rows and computes $\hat p_B - \hat p_A$ each time. For binary correctness, McNemar’s test goes even further: it discards rows where both models agree and keeps only discordant rows. If $n_{10}$ is the count where A is right and B is wrong, and $n_{01}$ where B is right and A is wrong, the null says each discordant row is equally likely to favor either model. The exact two-sided p-value is a binomial tail:

2\sum_{k=0}^{\min(n_{10},n_{01})} {n_{10}+n_{01} \choose k}2^{-(n_{10}+n_{01})} .\tag{8.4}

Listing 8.3 Paired bootstrap and exact paired tests

def paired_bootstrap_delta(a_correct, b_correct, draws=2000, level=0.95, rng=None):
    """Bootstrap interval for accuracy(B) - accuracy(A) on paired examples."""
    delta = np.asarray(b_correct, dtype=np.float64) - np.asarray(a_correct,
                                                                dtype=np.float64)
    rng = np.random.default_rng(0) if rng is None else rng
    n = len(delta)
    estimates = np.empty(draws, dtype=np.float64)
    for draw in range(draws):
        estimates[draw] = delta[rng.integers(0, n, size=n)].mean()
    alpha = (1.0 - level) / 2.0
    return delta.mean(), tuple(np.quantile(estimates, [alpha, 1.0 - alpha]))

def exact_permutation_p_value(a_correct, b_correct):
    """Two-sided sign-flip test for paired 0/1 correctness arrays."""
    delta = np.asarray(b_correct, dtype=int) - np.asarray(a_correct, dtype=int)
    signs = delta[delta != 0]
    observed = abs(signs.sum())
    if len(signs) == 0:
        return 1.0
    extreme = 0
    for wins_for_b in range(len(signs) + 1):
        total = 2 * wins_for_b - len(signs)
        if abs(total) >= observed:
            extreme += comb(len(signs), wins_for_b)
    return extreme / (2 ** len(signs))


def mcnemar_exact_p_value(a_correct, b_correct):
    """Exact two-sided McNemar p-value for discordant paired outcomes."""
    a = np.asarray(a_correct, dtype=bool)
    b = np.asarray(b_correct, dtype=bool)
    a_only = int(np.sum(a & ~b))
    b_only = int(np.sum(~a & b))
    n = a_only + b_only
    if n == 0:
        return 1.0
    tail = sum(comb(n, k) for k in range(min(a_only, b_only) + 1)) / (2 ** n)
    return min(1.0, 2.0 * tail)

In a tiny benchmark where B fixes four A errors and breaks one A success, the observed gap is 0.25 and both the exact permutation test and McNemar’s test give $p = 0.375$ . That is not evidence strong enough to trust the gap. If you inspect many models, prompt variants, metrics, or slices, adjust the story for multiple comparisons; one low p-value among many tries is easy to manufacture by chance [wasserman2004].

8.5 How many examples?

Before collecting evaluations, decide the smallest difference worth detecting. Rearranging (8.1) and (8.2) gives the sample-size arithmetic for a target half-width $h$ :

N \approx \frac{1.96^2\,p(1-p)}{h^2} .\tag{8.5}

The conservative choice is $p = 0.5$ , where $p(1-p)$ is largest. A 95% interval with half-width 0.02 then needs 2,401 examples. If you expect accuracy near 0.70, the same arithmetic needs 2,017 examples. Paired comparisons can need fewer examples when models make errors on the same rows, but the honest estimate then comes from a pilot set of per-row differences.

Listing 8.4 Sample size from a target half-width

def accuracy_sample_size(p, half_width, z=1.96):
    """Examples needed so a normal CI has about the requested half-width."""
    return int(np.ceil(z * z * p * (1.0 - p) / (half_width * half_width)))

In practice

Agent and coding benchmarks such as SWE-bench [jimenez2023swebench] and judge-based LLM comparisons such as MT-Bench and Chatbot Arena [zheng2023judging] are finite samples of a larger task population. Report intervals for the metric, not only a point estimate, and use paired tests when the same prompts are answered by both systems; the prompt is the natural blocking variable. Bootstrap rows, not tokens: the unit of evidence is usually the task, conversation, or user query. Keep a final test set untouched; repeated leaderboard peeking turns hypothesis testing into model selection.

Key equations

\hat p = \frac1N\sum_i Y_i

\widehat{\mathrm{SE}}(\hat p) = \sqrt{\frac{\hat p(1-\hat p)}{N}}

\text{normal CI} = \hat p \pm 1.96\,\widehat{\mathrm{SE}}(\hat p)

p\text{-value} = P_{H_0}(|T| \ge |T_{\mathrm{obs}}|)

N \approx \frac{1.96^2p(1-p)}{h^2}

8.6 Teach it

The one-sentence version: a benchmark score is a noisy sample mean, so compare models by the noise of the examples, not by wishful decimals. Analogy: judging a model from a benchmark is like judging a coin from flips; more flips shrink uncertainty, but never to zero. Board steps: write correctness as 0/1 variables; derive $\Var(\hat p)=p(1-p)/N$ ; draw the normal interval; for two models, replace raw scores with per-example differences. Misconceptions: 95% confidence does not mean a 95% chance this fixed interval contains the truth; a p-value is not the probability the null is true; unpaired tests waste information on paired benchmarks. Check for understanding: if two models answer exactly the same examples, what array should you bootstrap to estimate the uncertainty of their accuracy gap?

8.7 Exercises

Exercise 8.1 ★ Standard error

A model answers 248 of 400 benchmark examples correctly. Compute its accuracy, standard error, and 95% normal-approximation interval. Explain what the standard error measures.

Solution in Appendix D

Exercise 8.2 ★★ Bootstrap interpretation

Why can a percentile bootstrap interval be used for a median judge score even when the normal accuracy interval is not appropriate? State the resampling unit for an LLM benchmark.

Solution in Appendix D

Exercise 8.3 ★★ Paired test

On 12 examples, model B fixes four errors made by model A and breaks one example that A had right. The other rows agree. Compute the accuracy gap and the exact paired p-value.

Solution in Appendix D

Exercise 8.4 ★★★ Implementation

Write a function that returns the number of examples needed for a 95% normal interval with a chosen half-width. Evaluate it for worst-case accuracy and half-width 0.02.

Solution in Appendix D

References

[efron1979] B. Efron. Bootstrap methods: another look at the jackknife. Annals of Statistics 7(1), 1–26, 1979.
[mcnemar1947] Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157, 1947.
[wasserman2004] L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.
[jimenez2023swebench] C. E. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? 2023. arXiv:2310.06770
[zheng2023judging] L. Zheng et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. 2023. arXiv:2306.05685