The product loop
Discovery and delivery are no longer sequential phases. AI collapsed the build phase, and now the teams winning are the ones with the tightest loops.
Read articleJohan Rydberg
Blog
42% of Spotify experiments get rolled back. That's the point.
At Spotify, 42% of experiments are rolled back after guardrail metrics detect regressions. The discipline to discard is more valuable than the ability to ship.
Johan Rydberg
Accurate Sample Size for Always-Valid Inference
Most A/B testing tools either overestimate the sample size needed for always-valid inference or do not adjust for the sequential test in the sample size calculation at all. We derived a closed-form correction that requires no simulation.
Mårten Schultzberg
When A/B tests tell you what you want to hear
A/B testing maturity has a dangerous middle phase: teams test, dislike the answer, and explain it away. How to spot it and learn to trust your data.
Johan Rydberg
What experiments actually teach you
The primary output of a mature experimentation program is better judgment. At Spotify, the learning rate is 64%. The win rate is 12%.
Johan Rydberg
When AI writes the code, who decides what ships?
AI-accelerated code production increases the need for experimentation. The validation bottleneck grows with build speed. The fastest learners will win.
Johan Rydberg
What Makes a Good Sample Size Calculator?
A good sample size calculator must match your analysis: sequential testing, multiple metrics, and variance reduction all change the number it returns.
Mårten Schultzberg
The Judgment Gap
AI made building cheap. It also made bad decisions cheaper to ship. The distance between execution speed and validation speed is the judgment gap.
Johan Rydberg
The Real ROI of Experimentation
The ROI of experimentation goes beyond counting winners: shipped wins, prevented regressions, and faster organizational learning all add measurable value.
Johan Rydberg
Spotify's Experimentation Bootcamp is now free: Introducing Confidence Bootcamp
The A/B testing curriculum Spotify built over ten years to train thousands of experimenters is now free and open to everyone.
Mårten Schultzberg
All posts
Powered ≠ Trustworthy
Are Optimal Multiple Testing Corrections Optimal for You?
When Proxy Metrics Break: How Optimizing for Proxies Can Backfire
Why We Use Separate Tech Stacks for Personalization and Experimentation
Two Questions Every Experiment Should Answer
The Feature Flag Toolbox: Cloud, Edge, and Local
How Experimental Evidence Travels Through Your Organization: Why Better May Be Worse
Beyond Winning: Spotify's Experiments with Learning Framework
A/B Test Bandwidth: The Currency of Innovation
