Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 8: Set the sensitivity of the experiment with the minimum detectable effect (MDE)

Summary

In this lesson, you learn about the minimum detectable effect (MDE) and how you use it to set the sensitivity of an experiment.

The Minimum Detectable Effect (MDE):

Decides how small effects in your metrics you can detect.
The smaller MDE you choose, the larger the sample size you need for your experiment.

After deciding which metric to use to measure success, you need to define what effect size you want the experiment to be able to detect. This effect size is called the "minimum detectable effect" (MDE), or "minimum relevant effect". Use the MDE to set up the experiment so that it has enough sensitivity to detect meaningful effects.

Picking the MDE is a trade-off between:

the smallest effect relevant for the business
the smallest effect that's practically measurable with the sample size you can reach in your experiment

As an experimenter, you can use your domain expertise and discuss with stakeholders to decide what is the smallest effect that you would consider meaningful. In the next step, you calculate what sample size you need to be able to reliably measure this effect. If the sample size required to measure the chosen MDE is unrealistically large, then you need to adjust the MDE upwards.

One way to understand the minimum detectable effect (MDE) of an experiment is to imagine your experiment as a microscope.

Illustration: MDE as the resolution of a microscope

Imagine looking at tissue-sample under a microscope. The more you zoom in, the more details you can see. Changes to the sample that would be hard or impossible to see at one level of magnification become clear at a higher level of magnification. The minimum detectable effect (MDE) in an experiment is like the resolution of a microscope. It is the smallest change that you want to be able to see. If you want to be able to see smaller changes, you need a higher resolution. In experiments, you can increase the 'resolution' or sensitivity by increasing the sample size. The larger the sample, the smaller changes you can detect. Just like you cannot zoom in on a microscope indefinitely, you cannot detect arbitrarily small changes with an experiment, because you don't have an infinite number of users.

Learn about the Minimum Detectable Effect (MDE) and how to set it in your experiment in 3 minutes and 43 seconds.

Reader exercise

What is the purpose of the minimum detectable effect (MDE) in an experiment?

Determine the smallest effect that is relevant for the business and use it to inform about how many users the experiment needs to reach.

To set the right settings for microscope in any laboratory.

To minimize the detectable effect of the experiment to avoid mistakes.