Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 10: Run Your Experiment

Summary

In this lesson, you learn how to monitor your experiment after launch, including checking exposure counts, reviewing automatic monitoring checks, and accessing your results either continuously or once the experiment ends.

After you launch your experiment, you should monitor that everything works fine. A good experimentation platform supports you in this stage with automatic monitoring checks. In this section of the course you learn what those checks are, and how to monitor that your experiment delivers your new experience.

Directly after launch

When you launch the A/B test, exposure calculations run at frequent intervals initially. This makes it possible to directly see that the experiment works as intended.

In Confidence

Confidence runs automatic monitoring checks throughout the experiment. Hover the Live status on the right sidebar to see their current status.

If you use Confidence's flags, you can also see the real-time resolve count alongside the exposure schedule.

At any time while the experiment is live

As soon as the first exposure and metric calculation finish, you can check the status of monitoring checks for your experiment.

If you configured your experiment to deliver results continuously, the full results page is available while the experiment is live with results for all metrics that have at least some data.

When you end the experiment

If you configured your experiment to deliver results at the end of the experiment, the results page only shows the results for all metrics after you end it.

You can also use exploratory analysis to further drill down the results of metrics by splitting the results by dimensions, and add new metrics that weren't part of the initial experiment setup. You can, for example, split the results by market to see how results differ between different markets, or add a metric that measures something similar to an existing metric to make sure that an unexpected result is present also in the new metric.

You can use exploratory analysis during the experiment, but to prevent p-hacking and cherry picking of results, you should wait until the experiment has ended before exploring additional metrics.

In Confidence

Read more about how to interpret the results in Confidence. Use the Explore tab to split results by dimensions and add metrics beyond the initial setup.

Reader exercise

Which is a correct statement about monitoring and validation checks?

You have to use continuous results calculations to activate monitoring checks

Only rollouts have monitoring checks

Confidence alerts you if any validation check is failing for any of your experiments

Lesson 10: Run Your Experiment

Summary

Directly after launch

When you launch the A/B test, exposure calculations run at frequent intervals initially. This makes it possible to directly see that the experiment works as intended.

In Confidence

Confidence runs automatic monitoring checks throughout the experiment. Hover the Live status on the right sidebar to see their current status.

If you use Confidence's flags, you can also see the real-time resolve count alongside the exposure schedule.

At any time while the experiment is live

As soon as the first exposure and metric calculation finish, you can check the status of monitoring checks for your experiment.

If you configured your experiment to deliver results continuously, the full results page is available while the experiment is live with results for all metrics that have at least some data.

When you end the experiment

If you configured your experiment to deliver results at the end of the experiment, the results page only shows the results for all metrics after you end it.

You can use exploratory analysis during the experiment, but to prevent p-hacking and cherry picking of results, you should wait until the experiment has ended before exploring additional metrics.

In Confidence

Read more about how to interpret the results in Confidence. Use the Explore tab to split results by dimensions and add metrics beyond the initial setup.

Reader exercise

Which is a correct statement about monitoring and validation checks?

You have to use continuous results calculations to activate monitoring checks

Only rollouts have monitoring checks

Confidence alerts you if any validation check is failing for any of your experiments