Skip to main content
The sample size calculator helps you plan the length and size of an experiment by calculating what sample size you need to achieve the requested level of power.
When calculating the required sample size for an experiment, Confidence looks at historical data for the metrics in the experiment. There needs to be at least 14 days (plus the aggregation window and exposure offsets) of historical data for the metric. For example, if you have a metric with a 7-day aggregation window and a 7-day exposure offset, you need at least 28 days of historical data.

​
Calculate the Required Sample Size

1

Configure the experiment

Configure the experiment with your treatments, audience, and metrics.
2

Open the sample size calculator

In the Required sample size section on the right sidebar, click the top-right widget icon.
3

Click Calculate

Click Calculate on the widget to calculate the required sample size.
4

Review the results

When the calculation finishes, the widget displays a results table and a progress chart. The results table shows the following columns for each metric:
The Days needed, Expected samples, and progress charts are beta features. Contact experimentation-cs@spotify.com if you don’t see them.
  • Days needed: The estimated number of days to reach the required sample size.
  • Required samples: The sample size needed to achieve the configured level of power.
  • Expected samples: The projected total samples based on historical exposure rates.
  • Mean: The estimated mean of the metric from historical data.
  • Variance: The estimated variance of the metric from historical data.
The widget header shows the maximum days needed across all metrics. The progress chart above the table visualizes the estimated progress toward 100% of the required sample size for each metric over time. A vertical dashed line marks the current planned runtime.You can expand individual metric rows to view a per-metric chart that shows the expected and required sample sizes over time.

​
Define the Exposure Source

The exposure source is the source of the data used to calculate the mean and variance of the metrics.
1

Select the exposure source type

Choose one of the following:
  • Assignments: Use all existing assignments available in your assignment table, or filter these on assignments from specific flags
  • Previous experiment: Use exposure from a previous experiment
2

Filter assignments (optional)

If you selected Assignments, you can filter assignments on flags to only include a cohort of users similar to those in your upcoming experiment.

​
Adjust the Required Sample Size

If the required sample size is too large compared to the available population, you can either try to expand the population or reduce the required sample size. To reduce the required sample size, you can do one or more of the following:
  • Increase Alpha setting: Alpha is the probability of a false positive. A higher alpha requires a smaller sample size, but means the risk of finding significance when there really is no effect increases.
  • Lower Power setting: Power is the probability of a true positive. The higher the power, the lower the probability of a false negative. A lower power requires a smaller sample size, but lowers the chance of finding a true effect. Lower power also increases the risk of sign and magnitude errors (type S and type M errors). In general, a too low power makes it hard to reproduce the results of an experiment.
  • Increase metric MDEs and NIMs: The MDE and NIM are the effect sizes that you and your stakeholders care about. The larger the MDE and NIM, the smaller the required sample size.

Sample Size Calculator Reference

Deep dive into sample size calculations

Power Analysis

Understand statistical power

Effect Sizes

Learn about MDE and NIM settings

Filter Assignments

Improve sample size estimates