When AI writes the code, who decides what ships?
AI-accelerated code production increases the need for experimentation. The validation bottleneck grows with build speed. The fastest learners will win.
Read articleJohan Rydberg, General Manager
Blog
What Makes a Good Sample Size Calculator?
Most sample size calculators assume a fixed-sample test with a single metric. If your experiment uses sequential testing, corrects for multiple metrics, or applies variance reduction, the number is inaccurate and you are wasting resources.
Mårten Schultzberg, Staff Data Scientist
The Judgment Gap
AI made building cheap. It also made bad decisions cheaper to ship. The distance between execution speed and validation speed is the judgment gap.
Johan Rydberg, General Manager
The Real ROI of Experimentation
Most teams justify experimentation by counting winners. The real value shows up in three places: shipped wins, prevented harm, and how fast the organization learns from results.
Johan Rydberg, General Manager
Spotify's Experimentation Bootcamp is now free: Introducing Confidence Bootcamp
The A/B testing curriculum Spotify built over ten years to train thousands of experimenters is now free and open to everyone.
Mårten Schultzberg, Staff Data Scientist
Powered ≠ Trustworthy
Powering an experiment does not make its results trustworthy. Trustworthiness depends on whether the effect you powered for matches the true effect, and that is rarely something you can know in advance.
Mårten Schultzberg, Staff Data Scientist
Are Optimal Multiple Testing Corrections Optimal for You?
Bonferroni's conservatism reputation is mostly a denominator mistake. Here is why it holds up once you correct only the metrics that need it.
Mårten Schultzberg, Staff Data Scientist
When Proxy Metrics Break: How Optimizing for Proxies Can Backfire
Learn how wrong things can go when proxy metrics start to influence product development, and how to use them safely.
Mårten Schultzberg, Staff Data Scientist
Why We Use Separate Tech Stacks for Personalization and Experimentation
Why we maintain distinct technology stacks for personalization and experimentation.
Yu Zhao, Staff Machine Learning Engineer
Two Questions Every Experiment Should Answer
Learn how to never run an experiment without learning something.
Mårten Schultzberg, Staff Data Scientist
All posts
The Feature Flag Toolbox: Cloud, Edge, and Local
How Experimental Evidence Travels Through Your Organization: Why Better May Be Worse
Beyond Winning: Spotify's Experiments with Learning Framework
A/B Test Bandwidth: The Currency of Innovation
Experiments with Smaller Samples
Reduce Dilution and Improve Sensitivity with Trigger Analysis
Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at
Better Product Decisions with Guardrail Metrics
Collaboration Fuels Efficient Experimentation
