Want to experiment like Spotify? Sign up for a 30 day free trial.
Start your free trialMost people who run experiments are self-taught. They pieced the knowledge together from blog posts, internal wikis, chatbots, and trial and error. They know enough to launch an experiment. They often do not know enough to trust their results.
An underpowered experiment that reaches significance will almost certainly overestimate its effect. A sample ratio mismatch makes a result untrustworthy regardless of the p-value. A metric that invites gaming will be gamed. None of these are edge cases. They are the default failure modes for teams learning as they go.
During the last ten years, Spotify's experimentation team has been building, running, and revising this curriculum alongside the broader experimentation community at Spotify. Thousands of Spotify employees have learned experimentation from this material. Starting today, it is free and open to anyone.
Introducing the Confidence by Spotify Experimentation Bootcamp, now available for everyone
The Confidence by Spotify Experimentation Bootcamp is a self-paced curriculum covering everything from why randomized controlled trials work to cumulative holdback evaluations and sequential testing trade-offs. Eleven courses, 72 lessons, built on ten years of running and iterating on Spotify's experimentation program for data scientists, engineers, PMs, and leaders. No account required to start. Create a free account to save your progress.
Until today, this curriculum was available exclusively to paying Confidence customers. It is the same material we used to train experimenters across Spotify's thousands of employees, built and iterated on alongside the teams running tens of thousands of A/B tests every year. This is the curriculum that made that scale of experimentation possible.
The curriculum lives at confidence.spotify.com/bootcamp.
The curriculum starts before the statistics do
The first substantive lesson does not start with statistics. It starts with confirmation bias and the IKEA effect: we overvalue what we built ourselves, and we seek evidence that confirms what we already believe. Randomized controlled trials are the only reliable way around it.
Intro to Experimentation (12 lessons) covers the full A/B test lifecycle. Scientific Product Development (9 lessons) applies the scientific method to product decisions, including a case study on a real Spotify experiment. A Primer on Hypothesis Testing (6 lessons) covers the statistical mechanics with intuition rather than derivations. Intro to Metrics (10 lessons) covers metric design, the four metric roles, and variance reduction.
Sample Size Calculation spans three courses (13 lessons total), building from interactive intuition to multi-metric decision-making. Interpreting Experiment Results (11 lessons) covers the anatomy of a results page, sample ratio mismatch detection, and the winner's curse. Intro to Feature Flags (4 lessons) covers flag lifecycle, clients, and evaluation context. Experimentation Culture (3 lessons, 30 minutes) is for leaders and program builders. Advancing Experimentation (4 lessons) covers non-inferiority margins, sequential testing trade-offs, and cumulative holdback evaluations.
Who it's for
The bootcamp is for anyone who wants to run experiments well. The curriculum has role-based entry points for data scientists, engineers, PMs, and leaders, but the courses aren't gated by role. Anyone can start anywhere. At Spotify, the audience has spanned every role and function. Most courses contain no platform-specific UI or workflow; the concepts apply wherever experiments run.
The curriculum includes 21 interactive modules. In the hypothesis testing course, a sampling distribution simulator lets you slide the significance level and the observed mean difference and watch the p-value, critical value, and confidence interval all update in real time.
The sample size series builds from a basic interactive widget up to a full calculator where you set alpha, power, MDE, variance, treatment proportion, and the number of metrics under test — and it returns the required sample size alongside the full formula that produced it.
There are also confidence interval simulators, a Type M and Type S error explorer, and a false positive rate simulator. Seven animated illustrations walk through things like experiment randomization flow and how time windows change what a metric measures. A dedicated video library holds 13 short videos on topics from minimum detectable effect to feature flag targeting, each under five minutes.
Each lesson includes quizzes that check understanding as you go. Progress tracking works across all 11 courses with or without an account. Individual courses run between 30 minutes and 3 hours. You can finish any single course in one sitting.
Why we made the bootcamp free
Spotify's own experiments have a win rate of around 12%. The Experiments with Learning framework puts the learning rate at 64%: most value comes not from what you ship but from catching regressions before they reach everyone, invalidating bad ideas early, and answering questions that change what you build next. A neutral result from a well-powered experiment is worth as much as a win.
That philosophy only works if the people running experiments actually understand it. A platform alone doesn't build an experimentation culture. Knowledge does.
We've believed for a long time that the industry's experimentation quality problem is fundamentally a knowledge problem. Making this curriculum free is how we act on that belief beyond our own walls.
Ready to run the experiments, not just understand them?
The bootcamp is free and platform-agnostic. But if you want to put it into practice, Confidence is the experimentation platform built on the same decade of thinking behind this curriculum: warehouse-native, statistically rigorous, and designed to scale. Start a free 30-day trial and bring your first experiment to life.
Or start learning right now, no account needed, at confidence.spotify.com/bootcamp.