Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 9: Quality Assurance

Summary

To test whether your code works as intended, use override rules to assign specific users to your new feature. You can also run experiments on employees only, and run A/A tests to test your setup end-to-end before launching your main experiment.

Overrides

You can assign a specific user to a particular treatment by overriding the randomization. This means that you can add yourself or other members of the experimenting team to a specific variant at any time to try it out. You can verify that your implementations appear to be working as they should before releasing the experiment to actual users.

Overriding users into specific treatments doesn't affect the results, as the exposure data doesn't include the overrides.

In Confidence

In Confidence, you create override rules to assign specific users to a particular treatment.

Employee only

Depending on the nature of your product, a powerful next step in the QA process is to run your experiment on employees only. Make sure to include an attribute in the evaluation context that identifies the incoming request as belonging to an employee, and then use that in your inclusion criteria. This way, you can test your change and its different values on users that are a bit more forgiving. It can give you the chance to detect errors that you might not notice during the early stages of QA. The drawback is of course that the sample size is typically so small that it's difficult to find any meaningful effects, but you might hear from your colleagues if something isn't working as it should.

Note

For employee experiments to be possible you must include employee status in the evaluation context of your feature flag.

You can also give your new feature to employees only by directly creating a rule on your flag that has employee status as an inclusion criteria.

A/A tests

Sometimes you may want to run an A/A test to test your overall setup before launching the actual experiment. An A/A test is just like an A/B test, except that the experiences given to the control and treatment groups are the same. Either the two variants you use are the same, or you resolve the flag in your code but don't use the received variant values. A/A tests are particularly helpful if you want to test your whole setup end-to-end on real users and get real exposure data.

In Confidence

A/A tests are useful when you have just integrated your service with Confidence. The A/B test quickstart describes such a test.

Reader exercise

Whose responsibility it is to ensure that an experiment/rollout doesn't break the end-user experiment?

Statisticians and all other applied mathematicians

The experimenters, that is, all of us who are running experiments

The experimentation tool is responsible for everything

Lesson 9: Quality Assurance

Summary

Overrides

Overriding users into specific treatments doesn't affect the results, as the exposure data doesn't include the overrides.

In Confidence

In Confidence, you create override rules to assign specific users to a particular treatment.

Employee only

Note

For employee experiments to be possible you must include employee status in the evaluation context of your feature flag.

You can also give your new feature to employees only by directly creating a rule on your flag that has employee status as an inclusion criteria.

A/A tests

In Confidence

A/A tests are useful when you have just integrated your service with Confidence. The A/B test quickstart describes such a test.

Reader exercise

Whose responsibility it is to ensure that an experiment/rollout doesn't break the end-user experiment?

Statisticians and all other applied mathematicians

The experimenters, that is, all of us who are running experiments

The experimentation tool is responsible for everything