Why not Bayes?
Bayesian A/B testing is not one method. It is a family of configurations with fundamentally different statistical properties. The right question is not "Bayesian or frequentist?" but "which configuration delivers the guarantees your experiment program needs?" We believe most customers get everything they need from a high-quality frequentist offering. For Spotify, this has remained true for over a decade.
One label, three tiers of guarantees
Organize the common Bayesian configurations by what they guarantee and a hierarchy falls out. Each tier adds guarantees on top of the one below and costs more to operate.
Tier 1: Posterior coherence
Any proper posterior gives coherent belief updates and statements like "there is a 96% probability that B beats A." This remains valid regardless of when or why you stop, but it does not bound error rates or estimation precision. Posterior-probability and expected-loss stopping both sit in this tier.
Tier 2: Bounded FPR
Combine any proper prior with a Bayes factor stopping threshold and Ville's inequality bounds the false positive rate at α under continuous monitoring, no fixed horizon required. Frequentists know this exact statistic as the mSPRT behind always-valid inference. The "mixing distribution" a frequentist tunes for power is a prior by another name.
Tier 3: FDR and shrinkage
Add an empirical Bayes prior fitted to historical experiments to get FDR calibration and effect-size-aware shrinkage. Its point mass at zero estimates the fraction of metric-level observations that are null and absorbs the multiplicity correction. These guarantees depend on the prior being well calibrated.
Tier 1 is the most common default in commercial platforms, which makes the difference between tiers practical rather than academic.
False positive rates under continuous monitoring
Any experiment program produces false positives over time, regardless of whether its inference is Bayesian or frequentist. The question is whether the chosen configuration controls that rate. From the simulations in our paper (nominal α = 0.05).
- 30.3%
- False positives from flat-prior posterior-probability stopping. Under a flat prior, "P(B > A) > 0.95" is identical to "p < 0.05," so continuous monitoring inflates both. This is a Tier 1 configuration.
- ≈50%
- False positives from expected-loss stopping (ε = 0.02, flat prior). It ships whichever variant is ahead once the posterior concentrates. Also Tier 1 because the rule optimizes expected cost, not evidence against the null.
- ≤5%
- False positives from Bayes factor stopping, for every proper prior tested. This is a Tier 2 configuration. The guarantee is a frequentist bound, courtesy of the martingale property.
A flat prior is a z-test in a different vocabulary
Under a flat prior and the standard two-group normal model, the Bayesian outputs are the frequentist outputs. The posterior mean equals the maximum likelihood estimate. P(B > A) equals 1 − p. The 95% credible interval has the same endpoints as the 95% confidence interval. Analyzed once at a fixed horizon, this is frequentist inference under different notation. Same numbers, same conclusions.
Optional stopping is where the equivalence starts to matter. "Stop when P(B > A) > 0.95" is the same rule as "stop when p < 0.05", so monitoring it continuously inherits the same peeking problem. The posterior stays perfectly coherent the whole time, but coherence is a statement about beliefs, not about error rates.
The 0.95 threshold looks like it should mean α = 0.05. It does not. The α in "P(B > A) > 1 − α" is a decision parameter with no frequentist guarantee attached. It coincides with the error rate only in a fixed-horizon test under a flat prior. Under optional stopping the two quantities diverge, and the shared notation makes that easy to miss.
The stronger tiers, and what they cost to run
The connection between frameworks goes deeper than the flat-prior case. The mSPRT is exactly Bayes factor stopping under the same prior, and for most cost functions proposed in the A/B testing literature the optimal or near-optimal stopping rule is a Bayes factor threshold. The decision-theoretic, error-rate, and sequential-testing formulations converge on similar configurations. At Tier 2 the Bayesian-versus-frequentist debate is an argument about parameterization, not about methods.
Tier 3 is where Bayesian machinery earns properties frequentist defaults do not have, specifically false-discovery-rate calibration across a program and shrinkage that corrects the winner's curse. But it is earned, not free. The prior must be fitted to a representative corpus of roughly 200+ historical experiments, and the known failure modes cannot be fixed by collecting more data. Fit on a winner-selected corpus and the null-rate estimate collapses. Pool programs with different effect-size distributions and both inherit the wrong correction. Mix heterogeneous metrics and the prior is too wide for some and too narrow for others.
What Confidence runs instead
Group-sequential tests and always-valid inference, the frequentist sequential methods with the same unconditional error control that Tier 2 provides, plus Bonferroni correction across metrics and CUPED variance reduction. The choice between these and Bayes factor stopping is operational rather than philosophical. The guarantees are the same, and these are simpler to run.
Peek freely, by design
Group-sequential boundaries and always-valid intervals are built for continuous monitoring, so peeking is handled automatically. You can monitor results as they accumulate without invalidating the guarantees.
High power, controlled errors
In the paper's simulation, a daily-look GST achieved 99.6% power at a controlled 5% false positive rate, versus 89.9% for Bayes factor stopping with an oracle empirical Bayes prior. GST led on detection; oracle EB led on estimation accuracy.
No prior to maintain
No prior to choose, no corpus to curate, no null-rate estimate to keep calibrated as your program's effect sizes change over time. The statistical defaults are the same for your first experiment and your thousandth.
Multiple-testing corrections
Bonferroni is legible and independent of estimated non-null rates. Confidence also supports optimal multiple-testing corrections that optimally correct metrics based on their metric type. CUPED raises power without changing the error guarantees.
The longer story of how Spotify chose its sequential framework is in Choosing a Sequential Testing Framework on the Spotify engineering blog.
Where Bayesian methods are the right tool
The hierarchy ranks configurations by their guarantees, not by school. In several settings, Bayesian machinery delivers things the alternatives do not.
Estimation accuracy at scale
A well-calibrated empirical Bayes prior achieved the lowest estimation error of any configuration in the paper's comparison. Its shrinkage is a genuine cure for the winner's curse, and no frequentist default replicates it as neatly. If you have the corpus and someone who owns its calibration, that advantage is real.
FDR control through the prior
With Bayes factor stopping and a calibrated empirical Bayes prior, FDR control across metrics follows from the prior's null-rate estimate. This is a sophisticated multiplicity correction, not an absence of one, and it only works when the prior is well calibrated.
Explicit decision costs
If you can state the cost of a wrong launch, decision-theoretic stopping optimizes that objective. For most A/B testing cost functions, the near-optimal rule is a Bayes factor threshold. The approaches differ when the optimal policy is not, as with expected-loss stopping.
The method should follow from the risks your program needs to control, not from what the method is called. An organization minimizing expected cost per launch decision will land somewhere different from one bounding the false discovery rate across hundreds of experiments. Both can be right.
Questions and answers
Doesn't Bayesian testing solve peeking?
Some configurations do. The most common default does not. Under a flat prior, "stop when P(B > A) > 0.95" is algebraically the same rule as "stop when p < 0.05", and both produce a 30.3% false positive rate under continuous monitoring in simulation.
Bayes factor stopping genuinely solves peeking via a martingale argument that delivers a frequentist guarantee. There is no contradiction between the Likelihood Principle and bounding the false positive rate. With Bayes factor stopping you get both. The useful questions are whether you care about the false positive rate, and whether the configuration you select controls it.
Isn't "there's a 95% chance B beats A" easier to explain than a p-value?
It is, and that is the appeal. But under the flat prior most implementations default to, P(B > A) equals exactly 1 − p. Same number, friendlier phrasing. The catch is that the phrasing invites an error-guarantee reading and no such guarantee is attached at Tier 1.
We would rather report an interval whose guarantee survives the stopping rule than a probability that sounds stronger than it is.
What about expected-loss stopping and "risk" thresholds?
Expected-loss stopping optimizes a real objective, the expected metric cost of shipping the wrong variant. It is the right tool when shipping a null-effect variant is genuinely free. But under the null it ships the new variant in roughly half of all experiments once the posterior tightens.
In the paper's simulation, methods that require evidence before shipping achieve lower total regret as soon as the cost of a null-effect launch exceeds about 2% of a typical real effect. Maintenance, code complexity, and the roadmap slot a feature occupies are rarely free.
Would Confidence ever ship a Bayesian mode?
If we did, it would be a Bayes factor stopping based inference machinery. At Tier 2 the guarantees match what the platform already provides, so the question is whether a second mode of inference justifies its cost. In a platform where people collaborate on experiments and consume results across teams, two modes with different planning, monitoring, and outputs creates communication cost that must be weighed against the statistical benefit. We would nudge any customer to pick one framework and use it consistently, because the friction and fragmentation of running both tends to outweigh the marginal statistical gains.
The paper's bottom line is ours too. Start from the risks your program needs to control, and let the method follow.
Read more about Bayesian A/B testing
Why Spotify Is Not Using Bayesian A/B Testing
Why Bayesian A/B testing is a family of configurations—and why Spotify currently sees no need to add it alongside frequentist tooling.
Choosing a Sequential Testing Framework
A comparison of sequential testing methods and the reasoning behind Spotify’s use of group sequential tests.
Bayesian Inference Procedures for A/B Testing: An Overview
The paper behind this page: a systematic comparison of Bayesian configurations and their statistical guarantees.
Every number on this page comes from the simulations in the full paper. Read it on arXiv: Bayesian Inference Procedures for A/B Testing: An Overview. For a shorter companion, start with the Bayesian A/B testing glossary entry.