Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 5: Sample size calculation playground - level II

Summary

In this lesson, you build intuition for sample size calculations by using an interactive sample size calculator playground. By changing parameters that we have discussed in this course, you can see how the sample size changes.

This is the second level of the sample size calculation course, so there are still parameters that you can affect in a real experiment that are fixed here and cannot be changed.

The idea with this lesson is simple, you change the parameters in the sample size calculator and see how the sample size changes. In this lessons there are more questions than for usual lessons, and can be answered by trying things out with the sample size calculator.

Recommendation

Turn on the 'Show detailed formulas' option in the sample size calculator to see the formulas used to calculate the sample size. This can help you understand how the different parameters affect the sample size mathematically.

Required sample size calculator (level 2)

Metric parameters

Metric mean (µ)

Relative MDE/NIM (%)

Metric variance (σ²)

Statistical parameters

Alpha (false positive rate)

Power (true positive rate)

Experiment design

Number of group comparisons1

Number of success metrics1

Number of guardrail metrics with NIMs0

Number of guardrail metrics without NIMs (has no effect on sample size)0

Required Sample Size: 0

Show detailed formulas

Reader exercise

Does increasing the number of guardrail metrics with NIMs always increase the required sample size more than increasing the number of success metrics?

Yes, this is because guardrail metrics with NIMs increase the false positive rate.

Yes, this is because the formula makes the sample size increase exponentially with the number of guardrails.

No, because of the non-linearity of power and alpha in the formulas, which type of metric costs the most depends on the alpha and power settings.

This question doesn't have a yes or no answer.

Reader exercise

How do increasing the number of success metrics and increasing the number of comparisons affect the sample size?

Increasing success metrics has a larger effect than increasing comparisons

Increasing comparisons has no effect on sample size

They both increase the required sample size at the same rate

The effects depend on the correlation between metrics

Lesson 5: Sample size calculation playground - level II

Summary

This is the second level of the sample size calculation course, so there are still parameters that you can affect in a real experiment that are fixed here and cannot be changed.

Recommendation

Required sample size calculator (level 2)

Metric parameters

Metric mean (µ)

Relative MDE/NIM (%)

Metric variance (σ²)

Statistical parameters

Alpha (false positive rate)

Power (true positive rate)

Experiment design

Number of group comparisons1

Number of success metrics1

Number of guardrail metrics with NIMs0

Number of guardrail metrics without NIMs (has no effect on sample size)0

Required Sample Size: 0

Show detailed formulas

Reader exercise

Does increasing the number of guardrail metrics with NIMs always increase the required sample size more than increasing the number of success metrics?

Yes, this is because guardrail metrics with NIMs increase the false positive rate.

Yes, this is because the formula makes the sample size increase exponentially with the number of guardrails.

No, because of the non-linearity of power and alpha in the formulas, which type of metric costs the most depends on the alpha and power settings.

This question doesn't have a yes or no answer.

Reader exercise

How do increasing the number of success metrics and increasing the number of comparisons affect the sample size?

Increasing success metrics has a larger effect than increasing comparisons

Increasing comparisons has no effect on sample size

They both increase the required sample size at the same rate

The effects depend on the correlation between metrics