Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at
A new experimental design that lets you estimate sample size during the experiment without compromising inference.
Want to experiment like Spotify? Sign up for a 30 day free trial.
Start your free trialTL;DR
When you don't know the required sample size upfront, you have two options: run a sequential test or run an A/A test first. Sequential tests are typically less sensitive and introduce bias to the treatment effect estimator. A/A tests prolong experiments and don't guarantee accurate sample size calculations.
Fixed-power designs offer a third option. Not all forms of peeking inflate the false positive rate of fixed-sample inference. The key insight: you can peek at variance estimates to determine sample size without compromising your statistical inference—as long as you don't peek at treatment effects.
This approach lets you estimate required sample size during the experiment itself, adjust as user characteristics change, and still use standard non-sequential inference methods with correct coverage.
Read the full post on Spotify Engineering: Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at
Learn about experimentation from the free online Confidence bootcamp. This 72-lesson bootcamp has taught thousands of people at Spotify and other companies to experiment. Start learning right now!