Beyond Winning: Spotify's Experiments with Learning Framework
How the Experiments with Learning (EwL) framework measures experimentation success beyond win rates.
TL;DR
A successful experiment yields enough valid information to inform product decisions—not just those that find "winners."
At Spotify, the win rate across experiments is around 12%. But the learning rate is around 64%. Most of our learning doesn't come from wins. It comes from discovering what not to ship and detecting regressions before they reach users. The Experiments with Learning (EwL) framework captures this: an experiment is a learning if it produces a valid result that informs a product decision, whether that's a win, a detected regression, or a conclusive neutral result.
The framework helps identify improvement areas for teams and platforms, guides resource allocation, and drives innovation while avoiding bad product decisions.
Read the full post on Spotify Engineering: Beyond Winning: Spotify's Experiments with Learning Framework