Who is the Confidence Bootcamp for?

The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.

Is the bootcamp free?

Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.

The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.

How long does the bootcamp take to complete?

The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.

Do I need prior experience with A/B testing or statistics?

No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.

Who created the Confidence Bootcamp?

The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.

Lesson 5: Case study: Shuffle button in a shelf on Spotify Home

Write a hypothesis for an experiment that adds a shuffle button to a shelf on Spotify Home

This case study is based on an actual experiment at Spotify. The details have been modified for the purpose of this exercise. The experiment adds a shuffle button to the 'Try something else' shelf on the home screen of Spotify.

The text below gives some information on earlier research, theory, and motivation that led to an experiment which tested the impact of adding a shuffle button on the 'Try something else' shelf on the home screen.

Use the information below to formulate a testable hypothesis for this experiment.

Prior knowledge

In earlier user research, we found that Just play something actions decreased the perceived friction to start listening.
A large part of the consumption from Spotify is via surfaces that automatically select what audio to play.
Anecdotal evidence that the stress of having to choose can explain why users continue to consume the same content.

Theory about user need

Providing the user with low-effort paths in terms of making a decision can decrease the load on the user to start listening, and thereby increase engagement and audio consumption.

Motivation

If users get stuck in browsing for content, and are unable to make a decision, they might leave the app.
Offering a low-effort path to start listening will decrease perceived stress and can increase engagement and audio consumption from the home screen.
Similar features have been successful in other places in the app before.

Exercise: Write a testable hypothesis for this experiment

Which success metrics would you set for this experiment? Why would you choose these metrics?
What unintended side effects could you see when you add a shuffle button to this shelf? What guardrail metrics would you set to test for this?
Based on the information above, write a testable hypothesis for an experiment on a shuffle button on the Try something else shelf. You can use the template below as a starting point.

You can find answers to these questions on the next page.

Hypothesis template

Based on [prior knowledge], we believe that [theory about user need].

We think that [doing this/building this feature/creating this experience] for [these people/personas] will achieve [these outcomes].

We will know this is true when we see [metric results]. This will be good for [customers/artists/ our business] because [motivation].

Bonus questions

How much of an increase in these metrics would you want to see to call the experiment a success?
What is the largest decrease in the guardrail metrics that you would still consider acceptable?

You may not be able to give an exact number for questions 4) and 5), but think about how you would come up with a number to answer these questions.

Reader exercise

What is the primary reason for selecting 'Minutes played on week 1 after exposure' as a success metric instead of 'Consumption from the shelf with the shuffle button'?

To avoid inflated results from novelty effects, which are more pronounced for shelf-level metrics than for overall consumption.

To specifically measure the impact on overall audio consumption, aligning with the goal of increasing engagement.

Because shelf-level consumption metrics have higher variance than overall consumption metrics, requiring a larger sample size to reach significance.