# Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at

Last updated: 2026-05-12
Canonical source: https://confidence.spotify.com/blog/fixed-power-designs
Owner: Spotify AB
Authors: Mårten Schultzberg (Staff Data Scientist)

> If this file and the page at the canonical URL disagree, the page is authoritative.

A new experimental design that lets you estimate sample size during the experiment without compromising inference.

## TL;DR

When you don't know the required [sample size](/glossary/sample-size) upfront, you have two options: run a [sequential test](/glossary/sequential-testing) or run an [A/A test](/glossary/aa-testing) first. Sequential tests are typically less sensitive and introduce bias to the [treatment effect](/glossary/treatment-effect) estimator. A/A tests prolong experiments and don't guarantee accurate sample size calculations.

[Fixed-power designs](/glossary/fixed-power-design) offer a third option. Not all forms of peeking inflate the false positive rate of fixed-sample inference. The key insight: you can peek at variance estimates to determine sample size without compromising your statistical inference—as long as you don't peek at treatment effects.

This approach lets you estimate required sample size during the experiment itself, adjust as user characteristics change, and still use standard non-sequential inference methods with correct coverage.

**Read the full post on Spotify Engineering:** [Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at](https://engineering.atspotify.com/2024/05/fixed-power-designs-its-not-if-you-peek-its-what-you-peek-at)