Introducing Confidence Agent
Confidence Agent is an AI collaborator built into Confidence that works with your flags, experiments, metrics, and project documents.
Read articleSebastian Ankargren
Blog
The rise of the product builder
As AI increases individual leverage across the product development stack, the boundaries between PM, design, analysis, and engineering are blurring.
Johan Rydberg
The product loop
Discovery and delivery are no longer sequential phases. AI collapsed the build phase, and now the teams winning are the ones with the tightest loops.
Johan Rydberg
42% of Spotify experiments get rolled back. That's the point.
At Spotify, 42% of experiments are rolled back after guardrail metrics detect regressions. The discipline to discard is more valuable than the ability to ship.
Johan Rydberg
Accurate Sample Size for Always-Valid Inference
Most A/B testing tools either overestimate the sample size needed for always-valid inference or do not adjust for the sequential test in the sample size calculation at all. We derived a closed-form correction that requires no simulation.
Mårten Schultzberg
When A/B tests tell you what you want to hear
A/B testing maturity has a dangerous middle phase: teams test, dislike the answer, and explain it away. How to spot it and learn to trust your data.
Johan Rydberg
What experiments actually teach you
The primary output of a mature experimentation program is better judgment. At Spotify, the learning rate is 64%. The win rate is 12%.
Johan Rydberg
When AI writes the code, who decides what ships?
AI-accelerated code production increases the need for experimentation. The validation bottleneck grows with build speed. The fastest learners will win.
Johan Rydberg
What Makes a Good Sample Size Calculator?
A good sample size calculator must match your analysis: sequential testing, multiple metrics, and variance reduction all change the number it returns.
Mårten Schultzberg
The Judgment Gap
AI made building cheap. It also made bad decisions cheaper to ship. The distance between execution speed and validation speed is the judgment gap.
Johan Rydberg
All posts
The Real ROI of Experimentation
Spotify's Experimentation Bootcamp is now free: Introducing Confidence Bootcamp
Powered ≠ Trustworthy
Are Optimal Multiple Testing Corrections Optimal for You?
When Proxy Metrics Break: How Optimizing for Proxies Can Backfire
Why We Use Separate Tech Stacks for Personalization and Experimentation
Two Questions Every Experiment Should Answer
The Feature Flag Toolbox: Cloud, Edge, and Local
How Experimental Evidence Travels Through Your Organization: Why Better May Be Worse
