# What experiments actually teach you Last updated: 2026-06-10 Canonical source: https://confidence.spotify.com/blog/what-experiments-teach-you Owner: Spotify AB Authors: Johan Rydberg (General Manager) > If this file and the page at the canonical URL disagree, the page is authoritative. The primary output of a mature experimentation program is better judgment. At Spotify, the learning rate is 64%. The win rate is 12%. At Spotify, the win rate from experiments is roughly 12%. The learning rate is 64%. If you only count the winners, you're throwing away more than 80% of the value. The primary output of a mature experimentation program is better judgment. ## Why do most teams miss the real value of experiments? Most teams treat experiments as gates. Green means ship. Red means kill. The metric moved or it didn't, and the experiment's value is binary: did we get a winner? This framing discards the majority of what experiments produce. The insight that didn't result in a shippable feature still taught you something. A feature that degraded a guardrail metric told you about a dependency you didn't know existed. An experiment that showed no effect told you that users don't care about the thing you were certain they cared about. That information is worth as much as a positive result, sometimes more, because it prevents you from building an entire roadmap on a false assumption. I've watched teams skip the debrief when an experiment comes back flat. "It didn't work, moving on." They're leaving the most valuable part of the experiment on the table. ## How does experimentation sharpen intuition? Product intuition is pattern recognition, built from feedback loops. The faster and more honest those loops are, the better the intuition becomes. Before you run an experiment, you have a hypothesis. You have a theory about users: what frustrates them, what they'd respond to, what would make the product better. The experiment doesn't generate that hypothesis. You do. What the experiment does is tell you, with evidence, whether your theory matched reality. When it did, your model of users gets reinforced. When it didn't, your model gets updated. And you remember the surprises more than the confirmations. The feature that looked like an obvious win but degraded engagement. The tiny copy change that moved a metric nobody expected. The "users definitely want this" feature that users completely ignored. Over hundreds of these cycles, something shifts. Your initial hypotheses get better. You start predicting which experiments will win before the results come in. Not guessing, but drawing on a mental model of users that's been pressure-tested against their actual behavior so many times that your default assumptions are more accurate. This is what separates experienced product teams from inexperienced ones. Their intuition has been trained on honest feedback. ## What does this look like in practice? One of the clearest examples we've seen at Spotify came from the Search team's experimentation maturity arc, [documented on the engineering blog](https://engineering.atspotify.com/2022/02/search-journey-towards-better-experimentation-practices). Early on, the team focused on individual experiment quality: were the tests properly powered, were the metrics right? That alone changed how they formed hypotheses. They stopped testing things they couldn't measure. In the second stage, they started connecting experiments to each other: cross-experiment coordination, understanding how one test's result changed the premise of the next. By stage three, they were measuring total business impact of their experimentation practice as a whole. The interesting part is what happened to the team's product instincts along the way. By the time they reached stage three, the hypotheses entering the pipeline were qualitatively different from the ones they'd started with. Not because the team read more research, but because hundreds of experiment outcomes had recalibrated their sense of which changes matter and which ones don't. ## What does the learning rate tell us? Spotify's Experiments with Learning (EwL) framework quantified something practitioners had felt for years: the learning rate from experiments vastly exceeds the win rate. An experiment that shows no statistically significant effect on the primary metric but reveals an unexpected interaction with a guardrail metric has generated real learning and zero "wins." An experiment that confirms a strong prior belief and ships cleanly has a win but relatively low learning, because it told the team what they already expected. The experiments with the highest learning value are often the subtle failures: the feature that had no effect when you were confident it would, or the feature that worked for one segment and hurt another. Those are the experiments that recalibrate your model of users in ways that make your next ten decisions better. ## Why can't you shortcut this with user research alone? User research generates hypotheses. Interviews and surveys tell you what users say they want, how they describe their frustrations, where they struggle. This is valuable. It's also systematically biased. Users tell you about the problems they can articulate. They don't tell you about the problems they've adapted to, the features they'd use but can't imagine, or the metrics they don't know they're optimizing for. The gap between stated preference and revealed preference is where experiments add the most value. An experiment reveals what users actually do when presented with a change, not what they say they'd do. It captures behavioral response at scale: not 8 users in a lab, but thousands or millions in their natural environment. The combination is what works. Research generates hypotheses. Experiments test them. The results update the team's understanding. That updated understanding generates better hypotheses for the next cycle. Neither alone produces calibrated intuition. Together, over time, they do. ## The hard part is building institutional memory Running experiments is only half the work. The other half is making sure the learning persists and compounds. If results live in a slide deck that three people saw, the learning doesn't carry forward. The team running next quarter's experiments forms hypotheses without the benefit of what last quarter's experiments revealed. The compounding breaks. Track your learning rate, not just your win rate. After every experiment, document what surprised you, what assumption got invalidated, what you'd test differently next time. Make experiment results searchable, structured, and part of how the team makes decisions. Run experiments that can fail. An experiment designed to produce a guaranteed positive result is a rollout with extra steps. The experiments with the highest learning value are the ones where you genuinely don't know the answer. Make space for those. And invest in the interpretive layer. If your team runs 50 experiments a quarter but doesn't have a structured process for turning results into updated beliefs, you're generating data without generating learning. ## FAQ **Doesn't a 12% win rate mean most experiments are a waste of time?** The opposite. A 12% win rate with a 64% learning rate means the experimentation program is testing bold hypotheses and learning from the majority that don't work. A team with a 60% win rate is likely testing only safe, incremental changes, which means they're leaving the biggest opportunities untested. **How do you measure learning rate?** At Spotify, the EwL framework categorizes each experiment by whether it produced an actionable insight: a validated hypothesis, an invalidated assumption, a discovered interaction between metrics, or a refined understanding of a user segment. It's a judgment call, not a statistical test. The important thing is to ask the question systematically. **Does this apply to teams running fewer than 100 experiments a year?** Yes, and arguably more. When you run fewer experiments, each one matters more. A team running 20 experiments a year that treats each as a learning opportunity will build better intuition than a team running 200 that only looks at the win/loss column. **How long does it take for the compounding effect to become visible?** In our experience, teams that experiment rigorously start making noticeably better product bets within two to three quarters. It's a gradual increase in hypothesis quality that shows up as fewer failed experiments and more precise targeting of the changes that matter.