Skip to main content

Documentation Index

Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt

Use this file to discover all available pages before exploring further.

The sample size calculator is a tool that assists in planning the length and size of an experiment. The tool calculates what sample size you need to achieve the requested level of power given the set-up of the experiment. The required sample size differs across metrics. The tool also displays the largest required sample size across all metrics. Having a large enough sample size is important to ensure that the experiment has enough sensitivity to detect meaningful effects. For more about what affects the required sample size, see the power page. In addition to the required sample size, the calculator estimates the expected number of samples and the number of days needed to reach the required sample size for each metric. A progress chart visualizes this information, making it easier to plan the experiment’s runtime.
The expected sample size, days needed, and progress charts are currently in beta. If you don’t see these features in your organization, contact experimentation-cs@spotify.com to request access.
The sample size calculator doesn’t take audience targeting into account. If you are targeting a subset of the population, then the variance of the metrics might be different for different subsets of the population. Some subsets might have larger variance, which increases the required number of users to power a certain MDE/NIM, while others might have smaller variances which could then decrease the required sample size for a certain MDE/NIM.
Learn more about sample size calculations in the the sample size calculation course.

​
Sample Size for New Metrics

When calculating the required sample size for an experiment, Confidence looks at historical data for the metrics in the experiment. There needs to be at least 14 days (plus the aggregation window and exposure offsets) of historical data for the metric in order for Confidence to be able to calculate the required sample size. For example, if you have an experiment with a metric that has a 7-day aggregation window and a 7-day exposure offset, you need at least 28 days of historical data. If there is not enough historical data, Confidence can’t calculate the required sample size.

​
Expected Sample Size and Days Needed Beta

The calculator uses the exposure source to estimate how many samples per day the experiment can expect to receive. Based on this rate, it calculates:
  • Expected samples: The estimated total samples by the end of the planned runtime, based on historical exposure rates.
  • Days needed: The estimated number of days to reach the required sample size for each metric. The overall days needed shown in the widget header is the maximum across all metrics.
When the expected samples exceed the required samples, the experiment is on track to be sufficiently powered within its planned runtime. If the days needed exceeds the planned runtime, consider extending the runtime or reducing the required sample size.
Both expected samples and days needed are estimates based on historical exposure from the selected exposure source. The actual exposure in a live experiment may be lower if:
  • The experiment uses audience targeting that excludes some users who were included in the historical data.
  • The flag has other rules that route some traffic away from the experiment.
Consider these factors when planning the experiment’s runtime.

​
Progress Chart Beta

The progress chart shows the estimated progress toward the required sample size for each metric over time. The Y-axis displays the percentage of the required sample size that the experiment has accumulated, and the X-axis shows the number of days after the experiment starts. A horizontal line at 100% marks the required sample size target. A vertical dashed line indicates the current planned runtime. Lines that extend beyond this point appear as dashed projections based on the estimated daily sample rate. You can expand individual metric rows to view a per-metric chart. The per-metric chart shows the absolute sample size on the Y-axis instead of a percentage, with separate lines for the expected and required sample sizes.

​
Exposure Source

The required sample size calculation consists of three parts:
  • Obtaining an exposure source
  • Using the exposure source to calculate the mean and variance of the metrics
  • Calculating the required sample size for each metric based on the mean and variance
The exposure source is the source of the data used to calculate the mean and variance of the metrics. It can be one of the following:
  • Past assignments: Use all existing assignments available in your assignment table, or filter these on assignments from specific flags to only include a cohort of users similar to those in your upcoming experiment.
  • Previous experiments: Use exposure from previous experiments as an exposure source.

Calculate Sample Size

Step-by-step sample size guide

Power Analysis

Understand statistical power

Effect Sizes

Configure MDE and NIM settings

Statistical Settings

Configure alpha and power