Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at
A new experimental design that lets you estimate sample size during the experiment without compromising inference.
TL;DR
When you don't know the required sample size upfront, you have two options: run a sequential test or run an A/A test first. Sequential tests are typically less sensitive and introduce bias to the treatment effect estimator. A/A tests prolong experiments and don't guarantee accurate sample size calculations.
Fixed-power designs offer a third option. Not all forms of peeking inflate the false positive rate of fixed-sample inference. The key insight: you can peek at variance estimates to determine sample size without compromising your statistical inference—as long as you don't peek at treatment effects.
This approach lets you estimate required sample size during the experiment itself, adjust as user characteristics change, and still use standard non-sequential inference methods with correct coverage.
Read the full post on Spotify Engineering: Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at