The signal-to-noise ratio (SNR) in A/B testing is the ratio of the treatment effect (the signal) to the variability of the metric being measured (the noise). A higher SNR means the real effect is easier to detect; a lower SNR means you need more data, a bolder implementation, or better variance reduction to see what's actually happening.
SNR is the single concept that ties together most of the practical challenges in experimentation. When a team complains that their experiments "never reach significance," the underlying problem is almost always a low signal-to-noise ratio: either the treatment effect is too small, the metric is too noisy, or both. Understanding SNR clarifies why some experiments need millions of users while others need thousands, and why variance reduction techniques like CUPED can be the difference between a conclusive result and weeks of wasted runtime.
How does signal-to-noise ratio affect statistical power?
Statistical power is the probability that your experiment detects a real effect when one exists. Power is a direct function of SNR and sample size.
The relationship is straightforward. Power increases when SNR increases (stronger effect relative to noise) and when sample size increases (more data to distinguish signal from noise). A standard two-sample z-test achieves 80% power when the SNR, scaled by the square root of the sample size per group, exceeds roughly 2.8.
This means you have three levers for reaching adequate power:
Increase the signal. Ship a bolder implementation that produces a larger treatment effect. The Confidence blog calls this the "Maximum Viable Change": test the loudest version of the idea that still functions as a user experience, so you find out whether the lever exists before optimizing the implementation.
Decrease the noise. Apply variance reduction. CUPED typically reduces metric variance by 20-50%, which effectively multiplies your SNR by 1.2-1.4x. Metric capping handles the extreme tails. Trigger analysis removes users who never saw the change.
Increase the sample size. Run the experiment longer or on more traffic. This is the most expensive lever, because experiment bandwidth is a finite resource.
At Spotify, where teams run 10,000+ experiments per year on Confidence, the first two levers get priority. Running longer isn't free when experiment slots are the binding constraint on how fast the product improves.
What determines the noise in a metric?
The noise in an A/B test metric comes from natural variation in user behavior. Some sources of noise are inherent to the metric; others are artifacts of how the experiment is designed.
Metric variance. Heavy-tailed metrics (revenue, session duration, content consumption) have high variance because a small number of users generate extreme values. Binary metrics (converted vs. not converted) have variance bounded by p(1-p), where p is the base rate. A conversion metric at 50% has maximum variance; one at 1% has much lower variance but also a smaller absolute effect size.
Metric type. Ratio metrics and per-user averages tend to be noisier than simple counts. A metric like "revenue per active user" combines variation in revenue with variation in activity patterns.
Population dilution. If your experiment includes users who never encounter the changed feature, their data adds noise without adding signal. Trigger analysis solves this by restricting the analysis to exposed users.
Time window. Metrics measured over shorter windows are noisier per observation but accumulate less temporal autocorrelation. The right window depends on the metric and the decision you're making.
How can you estimate SNR before running an experiment?
You can estimate SNR during experiment design using historical data.
The noise component comes from the historical variance of the metric in your population. If you've measured "streams per user per day" across your user base, you know its standard deviation.
The signal component is harder. It requires estimating the minimum detectable effect (MDE): the smallest treatment effect you'd want to detect. The MDE should come from a product judgment about what effect size would actually change your decision. If a 0.1% improvement in conversion doesn't change whether you ship, don't power for it.
Confidence's power analysis tools combine these inputs to estimate the sample size needed for a given power level. When CUPED is enabled, the power calculation adjusts for the expected variance reduction, giving you a sample size estimate that matches what the analysis will actually do.