Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 4: Number of comparisons

Summary

This lesson teaches you about how the number of comparisons affects the required sample size in experiments. Everything else held constant, the more treatment groups and thereby comparisons you have, the smaller alpha you will have to use per metric to bound the false positive rate for the decision below alpha, which leads to a larger required sample size. The number of comparisons does not affect the power you need to use per metric.

The impact of multiple comparisons

The most common pattern in product A/B tests is to compare all treatment groups against a control group. This means there are as many comparisons as there are treatment groups being tested.

In principle, it is also possible to compare all treatment groups against each other. This would mean the number of comparisons equals the number of pairs of treatments.

The number of comparisons affects the required sample size. The more comparisons, the more samples are required. This is because the probability of making a Type I error (false positive) increases with the number of comparisons. To counter this, we adjust the alpha level for multiple comparisons, which increases the required sample size.

The intuition behind this adjustment is that the more tests we run, the more chances there are to find a significant result by random chance. For example, if we run an experiment with 100 treatments and an alpha of 10%, even if no treatment has any effect, we would expect to see 10 treatments with a (false positive) significant result just by chance.

Reader exercise

How does the number of comparisons in an experiment affect the required sample size?

The required sample size decreases with more comparisons

The required sample size increases with more comparisons

The number of comparisons doesn't affect the required sample size

The required sample size depends only on the power setting

Reader exercise

Why is the alpha level adjusted when there are multiple comparisons in an experiment?

To increase the probability of detecting true positives

To prevent the required sample size from decreasing

To bound the false positive rate for the overall decision below alpha

To ensure all treatment groups are compared equally

Notes for nerds

Some people wonder what to do if more than one treatment is significantly better than the control group. This is a deep question. You can test the treatments against each other to see if one is better than the other. However, the difference between the treatment groups is likely smaller than the difference between them and the control group. This makes the power to detect a difference between treatments lower than the power to detect a difference between a treatment and the control group.

There are more advanced methods for finding the best treatment among many, such as Tukey's and Scheffé's methods, and Dunnett's test. However, we don't recommend using these methods due to the complexity involved in learning how to use them. Instead, you should gather stakeholders to decide which of the significant treatments to implement based on factors such as:

Complexity
Cost
Future extensibility

Lesson 4: Number of comparisons

Summary

The impact of multiple comparisons

The most common pattern in product A/B tests is to compare all treatment groups against a control group. This means there are as many comparisons as there are treatment groups being tested.

In principle, it is also possible to compare all treatment groups against each other. This would mean the number of comparisons equals the number of pairs of treatments.

Reader exercise

How does the number of comparisons in an experiment affect the required sample size?

The required sample size decreases with more comparisons

The required sample size increases with more comparisons

The number of comparisons doesn't affect the required sample size

The required sample size depends only on the power setting

Reader exercise

Why is the alpha level adjusted when there are multiple comparisons in an experiment?

To increase the probability of detecting true positives

To prevent the required sample size from decreasing

To bound the false positive rate for the overall decision below alpha

To ensure all treatment groups are compared equally

Notes for nerds

Complexity
Cost
Future extensibility