42% of Spotify experiments get rolled back. That's the point.

At Spotify, 42% of experiments are rolled back after guardrail metrics detect regressions. Because the platform caught something the team didn't expect. In a world where building is cheap and ideas are plentiful, the discipline to discard is more valuable than the ability to ship.

#Why would a company roll back nearly half its experiments?

The number sounds alarming if you assume the goal of experimentation is to ship features. It makes perfect sense if you assume the goal is to build a better product.

Every feature that enters an experiment at Spotify has already passed internal review. Someone believed it would improve the product. A team invested time in building it. These are considered bets from people who know their users.

And still, 42% of those bets either didn't improve the target metric or degraded something else that mattered. The feature worked as designed but users didn't respond the way the team expected. Or users responded to the target change but their behavior shifted in an adjacent area that made the product worse overall. Or the effect was real but too small to justify the added complexity.

That's the normal outcome of building products for hundreds of millions of users whose behavior is complex, contextual, and frequently surprising. The failure would be shipping those features without catching the regressions.

There's an asymmetry here that gets sharper as a product matures. You've already optimized the UX for years. Users have habits and expectations. Finding a single improvement that moves the needle upwards gets harder over time. But it's always equally easy to introduce a regression: a bug that increases crashes, a layout change that slows load times, a flow that confuses a segment you weren't watching. The upside gets harder to find; the downside stays constant. With that asymmetry in place, avoiding bad decisions is as important as finding good ones.

#What does it mean to build in a world where building is cheap?

AI-assisted development has compressed the cost of building software. Features that took weeks now take days. Prototypes that required an engineering sprint now come together in an afternoon.

But most teams haven't fully internalized what this means for validation. When building was expensive, the cost itself filtered ideas. Teams couldn't build everything, so they debated, prioritized, and cut. The expense of execution created space for judgment.

Now, execution is cheap enough that judgment has to come from somewhere else. If it doesn't, more bad ideas make it to production: not because teams are less capable, but because the natural filter that used to kill bad ideas early has been removed.

If a team could ship 10 features per quarter before AI assistance and now ships 40, and their historical rate of features-that-actually-improve-the-product was roughly 50%, then before AI they shipped about 5 features that made the product worse. Now they ship about 20. Same quality of ideas, same rate of bad bets, just more of them.

The 42% rollback rate at Spotify is a consequence of deliberate discipline: every change that reaches users is measured, and the ones that don't improve the product are removed. Without that discipline, the same percentage of bad ideas would still exist. They just wouldn't be caught.

#What do guardrail metrics actually catch?

Guardrail metrics are metrics you define as things that must not get worse, separate from the metric you're trying to improve. They're the mechanism behind most rollbacks, and they work because product changes rarely affect only the thing you intended.

We see three patterns over and over.

A team optimizes a feature for engagement with a specific surface. The target metric improves. But users are spending time on that surface at the expense of another part of the product, and overall satisfaction decreases. Without a guardrail on the broader metric, the team sees a win and ships. With a guardrail, the regression is visible and the team can make an informed decision.

A change improves the average experience but degrades the experience for a specific segment. A guardrail scoped to that segment catches the regression even though the top-line metric looks positive.

A change has a positive short-term effect that reverses over time. Users engage with the novelty but revert to prior behavior within weeks. Without longitudinal guardrails, the team ships on the basis of the short-term signal.

These patterns are common, and Spotify's platform is built to detect them. That's where the 42% comes from.

#What happens when teams don't have this discipline?

The alternative to catching regressions isn't that regressions don't happen. They happen silently.

Without guardrail metrics, a team optimizes for their primary metric and ships changes that look positive in isolation but degrade the product in aggregate. Each individual change might be small: a 0.2% decrease in a metric no one was watching. But product quality is the sum of thousands of decisions, and small regressions compound.

The team is shipping constantly. Velocity metrics look healthy. A/B test win rates are respectable. But the product is slowly getting worse because the experiments that should have been rolled back were instead declared winners or pushed through without adequate measurement.

In a world where AI-assisted development lets teams attempt more changes per quarter, that catch rate becomes more important, not less. There are more opportunities for quiet regressions to slip through.

This is why automating regression checks matters. At Spotify, guardrail metrics aren't optional add-ons that experimenters choose to include. They're required: both global guardrails (metrics that every experiment on the platform must monitor, like app stability and core engagement) and local guardrails (metrics specific to the surface or feature being tested). The platform enforces them automatically. An experimenter doesn't need to remember to check for regressions. The system checks for them and surfaces the result. When you're running 10 new experiments per week on a single surface, manual regression checking breaks down. Automated, required guardrails are what make the 42% catch rate possible at scale.

#The hard part is cultural

The lesson: the discipline to discard is a capability you have to build deliberately. It doesn't emerge naturally from running experiments.

Define guardrail metrics before you run the experiment, not after. If you wait until you see the results to decide what constitutes a regression, you'll find a way to explain away every one. Guardrails are commitments, not post-hoc rationalizations.

Normalize rollbacks. A culture where rolling back a feature is treated as a failure is a culture where regressions get shipped. The 42% rollback rate at Spotify is a sign the experimentation program is working as designed.

Set your guardrails to catch real regressions, not statistical noise. A guardrail that fires on every experiment is too sensitive. A guardrail that never fires isn't measuring the right thing. The right calibration comes from understanding the inferiority threshold: the minimum regression that's actually meaningful for your product and your users.

And separate the decision to test from the decision to ship. Running an experiment is a commitment to learn. If your team assumes every experiment will result in a launch, the incentive to interpret results honestly is undermined from the start.

#FAQ

Is 42% an unusually high rollback rate? It's higher than most reported industry benchmarks, but the comparison is misleading. Many companies report win rates that include experiments with no guardrail metrics, underpowered tests interpreted as positive, or experiments not designed to detect regressions. Spotify's 42% reflects a program where guardrails are standard, power is adequate, and regressions are systematically detected. A more useful question: what percentage of your shipped features would have been rolled back if you had the measurement infrastructure to detect regressions?

Does this mean Spotify wastes engineering effort on features that don't ship? Building a feature that gets tested and rolled back generates knowledge about what users want or don't want, which informs the next set of decisions. The wasted effort is building a feature, shipping it without testing, and never knowing whether it made the product better or worse.

What if our team doesn't have enough traffic to detect guardrail regressions? Variance reduction techniques like CUPED (a technique that uses pre-experiment data to tighten confidence intervals) can reduce required sample sizes by ~50%. Sequential testing lets you stop early when evidence of a regression is clear. And for teams with genuinely small sample sizes, even a less-powered test that detects large regressions is better than no measurement at all.

Should we roll back features where the primary metric is positive but a guardrail is negative? This is a judgment call. The guardrail regression should trigger investigation: how large is the regression? Is it transient or persistent? Is the guardrail metric causally connected to user value, or is it a proxy that might not matter? Guardrails force the conversation. They don't automate the decision.

How does this change in an AI-assisted development world? It becomes more important. When teams can build more features faster, the volume of changes that needs validation increases. If your team adopted AI coding tools but didn't scale its experimentation infrastructure, the percentage of unvalidated changes in production is likely growing.