Lesson 4: Sequential tests and sample size

Why use sequential tests?

A standard fixed-sample test collects all the data before looking at the results (see Calculation frequency for the basics). If the effect is large (which you won't know during the experiment), you still wait until the planned sample size is reached. Sequential tests let you check the results during the experiment and stop early when the evidence is clear. This means faster decisions and, on average, less data spent per experiment.

In this lesson we focus on the sample size implications of sequential testing.

The sample size trade-off

The required maximum sample size for a sequential test is larger than for a fixed-sample test. This is somewhat confusing: the average sample size can be lower, but we need a larger maximum sample size to guarantee power. Let's break this down.

The test boundaries must be wider to account for the multiple opportunities to reject the null hypothesis. Wider boundaries mean you need a larger planned sample size to achieve the same power. But power is the probability of rejecting at any point during the experiment, and most rejections happen before the maximum. So the planned sample size goes up, but depending on the type of sequential test, the average time to a decision can go down.

The power of a sequential test is not the power at the final analysis. It is the probability of rejecting the null hypothesis at any of the analysis points, given that the true effect equals the minimum detectable effect. Because each interim analysis is an opportunity to reject, the overall power is the sum of the rejection probabilities across all looks. A proper sample size calculation for a sequential test must account for this by integrating over all interim looks, not just computing the power at the last one.

The amount of extra data depends on the type of sequential test and how it is configured.

Group sequential tests

A group sequential test (GST) divides the experiment into a fixed number of interim analyses. At each analysis, the test statistic is compared against a boundary. If the statistic crosses the boundary, the experiment can be stopped early. If not, data collection continues to the next analysis.

The boundaries are chosen so that the probability of ever crossing them under the null hypothesis equals the desired significance level. The most common approach is alpha spending: the total allowable false positive rate is distributed across the interim analyses according to a spending function.

Different spending functions produce different boundary shapes:

• O'Brien-Fleming: spends very little alpha early on, with boundaries that decrease over time. This makes it hard to stop early unless the effect is very large, but it preserves nearly all the power of a fixed-sample test. The sample size inflation is typically just 1-5%.
• Pocock: spends alpha more evenly, with constant boundaries at each analysis. This makes early stopping easier but requires more data overall.

The number of interim analyses also matters. More analyses give more opportunities to stop early, but each additional look requires slightly wider boundaries, increasing the total sample size.

Always-valid inference

Always-valid tests (also called anytime-valid tests) take the sequential idea to its logical extreme: instead of a fixed number of analyses, you can check the results at any point during the experiment, as often as you like, without any pre-specified schedule. The false positive rate remains controlled regardless of when or how often you look.

This flexibility is powerful, but it comes at a steeper cost. Because the test must protect against an unlimited number of future looks, the boundaries are wider than those of a group sequential test. The result is a larger sample size requirement for the same statistical power.

Using properties of Brownian motion, the sample size overhead of an always-valid test relative to a fixed-sample test can be well approximated by a closed-form multiplicative factor, k*, that depends only on the significance level and the desired power:

n(always-valid) = k* × n(fixed)

At common settings (α = 0.05, power = 80%), k* is typically between 2 and 3 depending on the effect size, meaning an always-valid test needs roughly 2-3 times as much data as a fixed-sample test.

How the approaches compare

The interactive chart below shows how the sample size inflation factor varies across the two approaches. To simplify the examples, we assume linear traffic intake so that interim analyses are equally spaced in information fraction. The GST inflation is computed for a handful of representative K values and interpolated, so the line is approximate. Group sequential inflation depends on the number of interim analyses (K), while the always-valid factor k* is a fixed overhead that does not depend on K.

A few patterns to notice:

• GST inflation is small, especially with O'Brien-Fleming boundaries. Even with 100 interim analyses, the overhead is only a few percent.
• Always-valid inflation is much larger, typically requiring 2-3 times the fixed-sample size depending on alpha, power, and effect size. This is the price of unlimited flexibility.
• Higher power means less relative overhead for both methods. At 95% power, the always-valid factor drops to around k* ≈ 2.2.
• More interim analyses increase GST inflation, but the effect is modest: going from 2 to 100 analyses adds only a few percentage points.

What this means for your experiment plan

When deciding between sequential and non-sequential testing, the sample size implications are:

1. Fixed-sample test: no overhead, but you cannot act on interim results.
2. Group sequential test: small overhead (1-5% with O'Brien-Fleming), and you can stop early if the effect is large. This is usually the best trade-off when you have a reasonable expected sample size.
3. Always-valid test: large overhead (2-3x the fixed-sample size at typical settings), but you get complete freedom in when and how often you evaluate.

You can also think about it the other way around: if you keep the sample size fixed and switch to a sequential test, you are trading statistical power for the ability to stop early. Running a sequential test at the fixed-sample size without adjusting for the overhead means running an underpowered experiment. This increases the risk of the winner's curse: when an experiment stops early because it crossed a wide boundary, the point estimate at that moment tends to be inflated, because only unusually large estimates cross the boundary early.

For most experiments where you have an estimated sample size, group sequential tests offer the best of both worlds: the ability to stop early with minimal sample size cost.

Notes for nerds

Burn-in and batched data

The always-valid overhead depends on the burn-in period: the earliest point at which the test starts monitoring. A later burn-in produces tighter boundaries and a smaller $k^*$, but delays the first chance to reject. There is an interesting practical tension here. Most experimentation platforms process data in daily batches rather than continuously, so you rarely benefit from monitoring in the first few hours anyway. This means a larger burn-in is often practical at no real cost, and the tighter boundaries it produces can meaningfully reduce the total sample size. On the other hand, if you only check results once per day, the true power is lower than the theoretical continuous-monitoring power, since you miss crossing events between batches.

Group sequential power integral

The required sample size for a group sequential test is the smallest $N$ such that the overall power equals the target. Under drift $\theta$, the test statistics at $K$ analyses follow a multivariate normal with correlations determined by the information fractions. The overall power is (Jennison & Turnbull, 1999):

$\pi(\theta) = \sum_{k=1}^{K} \int_{c_k}^{\infty} g_k(z; \theta) \, dz$

where $g_k$ is the sub-density of the test statistic at look $k$, conditional on not having crossed any boundary earlier. The recursion is:

$g_1(z; \theta) = \phi\!\left(z - \theta \sqrt{\tau_1}\right)$

$g_k(z; \theta) = \int_{-\infty}^{c_{k-1}} g_{k-1}(w; \theta) \cdot \frac{\phi\!\left(\frac{z - w\sqrt{\tau_{k-1}/\tau_k} - \theta(\sqrt{\tau_k} - \sqrt{\tau_{k-1}})/\sqrt{\tau_k}}{\sqrt{1 - \tau_{k-1}/\tau_k}}\right)}{\sqrt{1 - \tau_{k-1}/\tau_k}} \, dw$

Here $\tau_k = I_k / I_K$ are the information fractions at each analysis and $\phi$ is the standard normal density. The boundaries $c_1, \ldots, c_K$ are determined by the alpha spending function under the null ($\theta = 0$). For O'Brien-Fleming-type spending, this uses the Kim-DeMets spending function $\alpha^*(t) = \alpha \cdot t^\rho$ with $\rho = 2$ (Lan and DeMets, 1983).

The key insight is that the Markov property of Brownian motion reduces the $K$-dimensional integral to a sequence of one-dimensional integrals, which can be solved efficiently via numerical integration or FFT methods.

The interactive examples in this lesson approximate the GST inflation via interpolation from a pre-computed grid. For exact values, use the Confidence sample size calculator.

Always-valid $k^*$

The key insight behind the closed-form $k^*$ is that the power of an always-valid test can be expressed as a boundary-crossing probability for a Brownian motion with drift. By linearizing the confidence sequence boundary at the candidate sample size, this crossing probability reduces to a bivariate normal CDF, which can be evaluated exactly. The full derivation is in Schultzberg (2025).

For a one-sided test at level $\alpha$ with power $1 - \beta$, the fixed-sample drift is $\mu_d = z_\alpha + z_\beta$. The always-valid test using a confidence sequence boundary $b(t)$ approximately achieves power $1 - \beta$ at sample size $k^* \cdot n_z$, where $k^*$ is the smallest $k > t_0$ satisfying a boundary-crossing probability equation.

For the boundary used in Confidence, the tuning constant is $\lambda_M = -W_{-1}(-\alpha^2/e) - 1$, where $W_{-1}$ is the lower branch of the Lambert $W$ function (Maharaj et al., 2023). The boundary at rescaled time $t$ is:

$b_M(t) = \sqrt{\frac{2(\lambda_M t + 1)}{\lambda_M} \log\!\left(1 + \frac{\sqrt{\lambda_M t + 1}}{2\alpha}\right)}$

The crossing probability at $k$ involves a bivariate normal CDF (Drezner and Wesolowsky, 1990) and has the form:

$\pi(k) = \Phi(-c_x) + \Phi_2\!\left(c_x,\, \frac{\nu T - \sqrt{t_0}\, c_x}{\sqrt{T + t_0}};\, \rho\right) + e^{2\sqrt{t_0}\,\nu(c_x + \sqrt{t_0}\,\nu)} \, \Phi_2\!\left(c_x + 2\sqrt{t_0}\,\nu,\, \frac{-\nu T - \sqrt{t_0}\, c_x - 2t_0\nu}{\sqrt{T + t_0}};\, \rho\right)$

where $L_{t_0} = b(k) - b'(k)(k - t_0)$, $c_x = (L_{t_0} - t_0 \mu_d)/\sqrt{t_0}$, $\nu = \mu_d - b'(k)$, $T = k - t_0$, and $\rho = -\sqrt{t_0}/\sqrt{T + t_0}$. This is computed exactly in the interactive examples above.