Accurate Sample Size for Always-Valid Inference

If you use always-valid confidence sequences for your experiments, your sample size calculator is probably telling you to run longer than you need to.

The standard approach, sometimes called the last-point heuristic, takes the fixed-sample calculation and inflates it until the test has the right rejection probability at the planned endpoint. But this ignores what makes sequential tests sequential. They can reject early. A test that has a 74% chance of rejecting at the endpoint might have 80% total power once you account for all the interim looks where the test statistic already crossed the stopping threshold. The last-point heuristic doesn't see those early rejections, so it keeps inflating.

This is the standard approach in the industry. Platforms that do adjust their sample size calculator for sequential testing, such as GrowthBook and Eppo, use variants of it. Many other platforms do not adjust for sequential testing in the calculator at all.

The heuristic was designed to be conservative as a way to sidestep the harder problem of computing power under optional stopping. In our simulations, targeting 80% power delivers 86-88% instead. That is 7-9 percentage points you didn't ask for, paid for with 8-20% larger samples than you actually need. It bounds the power, but it wastes your traffic.

A closed-form fix

We wanted a correction factor that gives you the right sample size without running simulations every time someone adjusts the effect size or significance level in a calculator. Concretely, we wanted to find a multiplier k* such that k* · n_fixed ≈ n_sequential, where n_fixed is the standard fixed-sample size and n_sequential is the sample size needed for the sequential test to reach target power.

Using Brownian motion approximations and a linear approximation of the boundary near the endpoint, we found one. The correction depends on the boundary only through its value and slope at the planned endpoint, which means it works for any smooth concave boundary, including WSKR confidence sequences, Maharaj confidence sequences, and the mixture sequential probability ratio test (mSPRT). Since the mSPRT is a Bayesian method, the correction covers Bayesian stopping rules too.

At the most common settings (alpha = 0.05, 80% power) with a realistic burn-in of 20 observations per arm, the corrected multiplier on the fixed-sample size is 2.50 instead of 2.90 for WSKR boundaries. That is a 14% reduction in required sample size. Across the full grid of significance levels, power targets, and boundary types, the savings range from 8% to 20%.

We validated the correction in simulation across Gaussian outcomes and allocation ratios. Empirical power lands within approximately 3 percentage points of target across the grid.

What it looks like in production

We also measured what the correction delivers on Spotify's actual experimentation platform, across 713 metrics from the last 283 experiments using always-valid inference.

The median saving was 9.5%. Lower than the theoretical 14%, because in production we apply Bonferroni corrections for multiple comparisons. Tighter significance levels mean the stopping boundary is harder to cross early, so the share of power from early rejections is smaller and the heuristic's error shrinks. This is an honest number from a real system.

The approximation degrades for metrics with very low base rates (below 1%). Those are typically regression-monitoring metrics, not the metrics you're powering experiments to detect. For the metrics that drive your sample size calculation, the correction holds.

Smaller samples mean more concurrent experiments

On Spotify's mobile home screen alone, 58 teams ran 520 experiments in one year, averaging 10 new experiments every week. When experiments compete for the same user population, sample size is what determines how many can run at once.

If an experiment allocates 5% of users and you reduce its required sample by even 10%, you free half a percentage point of your user base. Across a portfolio of concurrent sequential experiments, that can free a slot for one or two more concurrent experiments. More experiments means more chances to learn, and at Spotify's ~12% win rate, more experiments per quarter means more chances to find what works.

The Confidence sample size calculator uses this correction factor to more accurately represent the required sample size for always-valid testing, just like it accounts for group sequential testing when that method is used.

Go deeper in the bootcamp

The Confidence Bootcamp now covers this topic in depth. The sequential testing and sample size lesson walks through why sequential tests require larger maximum sample sizes, how group sequential tests and always-valid inference compare, and where the correction factor comes from. The sample size playground now supports all three paradigms: fixed-sample tests, group sequential tests, and always-valid inference, so you can see how the required sample size changes as you switch between them.

We run our decisions on this

We publish the research behind our tooling because we run our decisions on it. When the sample size formula is wrong, Spotify pays the cost in slower experiments and lower throughput.

Confidence handles sample size properly for group sequential tests (GST) too. The required sample size for GST has an analytical expression that can be evaluated using numerical integration, so the answer is exact rather than approximated. For always-valid inference no such expression exists, which is why the closed-form correction factor matters. The full derivation, simulation results, and production validation are in the paper.