Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 5: Experiment setup

Summary

This lesson goes over the main steps of setting up an experiment.

After the planning stage, you need to translate the details of your plan into an actual experiment configuration. At this stage you create an experiment and configure it according to your plan, which includes:

Configure the test variants: What part of the user experience will the test change for the users?
Select your target audience: For which users will the experience change?
Select the metrics you want to track: What are the key success metrics you want to measure? What are important guardrail metrics?

Configure the test variants

To set up an experiment, you need to define the test variants and connect the variants to the product code. Feature flags are the standard mechanism for connecting experiment variants to your product.

Watch this video to get a quick understanding of how feature flags let Confidence control parts of your product remotely.

In Confidence

To get started with feature flags in Confidence, follow the feature flag quickstart. If you want to go through the full steps of setting up an experiment, follow the A/B test or Rollout quickstarts. These guides have onboarded thousands of experimenters at Spotify and are a great way to get started.

Select your target audience

When setting up an experiment, you need to define the target audience for the experiment. This is the group of users that will be included in the experiment. You can define the target audience based on different user attributes, such as country, platform, or user age. What you can target on is determined by what information is passed in when you resolve the feature flag.

Watch this video to get a quick understanding of how targeting and feature flag resolving are related.

Select metrics

Refer to Lesson 3 and Lesson 4 for details on the types of metrics and how to select them.

Sample size and design

One important aspect of the experiment setup is the required sample size. The required sample size is the number of users that you need in your experiment to reach a level of precision in your results that lets you answer the question you have set out to answer. You can control the sample size by setting the allocation, the proportion of the target population that your experiment will include.

Learn more about sample size calculations in Lesson 8.

Reader exercise

What does the allocation control?

The proportion of experiments with a false result.

The proportion of the target population that your experiment will include.

The proportion of the target population that will not be in your experiment.