Bayes vs Frequentism: Goals All the Way Down

Background: We released a blog post and a paper on our view on Bayesian A/B testing. Michael Leznik wrote a thoughtful response that we really liked and wanted to respond to here.

Michael, thanks for the careful reading. Your response triggered many interesting thoughts and discussions on our end. Some of your points raise discussions we undervalued in the paper, and we aim to address those in a new version. Others we think are arguments about which goal matters, or about which estimand and modelling assumptions make sense for that goal, rather than about which framework delivers it. We will try to be precise about those distinctions in this response and analyze each criticism in turn.

#The likelihood

“The likelihood is listed, then dropped.”

Michael Leznik, “Model-First, Not Bayes-First”

Michael argues that good Bayesian practices, like Kruschke's BEST framework, take the likelihood seriously. Its Student-t likelihood can lead to different conclusions from a Gaussian model on the same data, especially for heavy-tailed metrics.

We agree that the paper says too little about the likelihood. But we do not see the difference between our setup and Kruschke's as a gap in our treatment of Bayes. It is a modelling choice within the configuration, presented as part of the Bayesian framework in Kruschke's account but available to both Bayesians and frequentists.

Whether reducing the influence of extreme observations is desirable depends on the goal and estimand. If you want a robust notion of location, it may be exactly right. If you want the literal arithmetic mean of an uncapped revenue distribution, it may instead change what the model is effectively estimating in contradiction with your goals.

This is why we separate the modelling question from the framework question. You choose an estimand for the goal, then a configuration whose assumptions support inference about it. A robust likelihood can be appropriate under either framework, just as a Gaussian working model can be when the target is a mean and the sampling approximation is adequate.

#HDI and ROPE

“The Bayesian configuration under attack is the weakest one.”

Michael Leznik, “Model-First, Not Bayes-First”

Michael argues that we simulated a weak Bayesian configuration while ignoring Kruschke's recommended HDI+ROPE approach, which can accept the null for practical purposes. From our perspective, this is mainly a disagreement about which goal matters, not which framework can achieve it.

If the goal is to accept practical equivalence, that goal should be stated explicitly. This approach is one Bayesian configuration for doing so. TOST, non-inferiority bounds, bootstrap, and related frequentist procedures provide other routes. They are not automatically equivalent, and comparisons based on common error rates or sequential monitoring require calibration. But the underlying three-way decision, positive effect, practical equivalence, or insufficient evidence for either, is not uniquely Bayesian.

To us, the omission is therefore not a missing Bayesian configuration for the goals we studied. We focused on a narrower set of programme goals common in product experimentation, and practical equivalence was not among them. Your criticism is useful because it highlights that the paper makes that scope look more exhaustive than it is. HDI+ROPE is a configuration for an additional goal that we could have named explicitly.

#Bayes factors

“The recommended configuration is the one estimation distrusts.”

Michael Leznik, “Model-First, Not Bayes-First”

Michael argues that Bayes factor (BF) stopping, which we present as a natural Bayesian configuration, is precisely what the estimation school distrusts. A BF can favour the null while the posterior remains widely uncertain, and it can be more sensitive to the prior than posterior inference.

We agree that the BF and posterior can point in different directions, and that this is important. But treating that disagreement as a failure of the BF presumes a particular goal. A configuration cannot be judged as good or bad without specifying the goal, and the information shown on a results page must also be aligned with that goal. We return to the latter in the interpretation section.

For several goals we study, the BF arises naturally as part of the configuration. In the simple-null setting considered here, it provides a sequential upper error guarantee, making it a natural choice when the goal is to bound false-positive rates under continuous monitoring. From a decision-theoretic perspective, BF thresholds are also optimal sequential rules under certain loss functions that arise naturally in product experimentation.

In both cases, the alternative prior matters, and Bartlett’s paradox is relevant. The BF’s sensitivity to prior choice is important and arguably underdeveloped in the paper. On this point, Michael's argument and Kruschke’s are well aligned with ours. Prior selection is an underestimated challenge, and finding the right prior may be difficult. We should discuss more clearly that this sensitivity can be stronger for the BF than for posterior inference.

This said, from our perspective, the choice should never be framed as posterior inference versus BF inference. The goal should determine the configuration, not the other way around. If a BF-based configuration can deliver the goal but its validity is too sensitive to prior choice, the answer is to find another configuration that delivers the same goal more robustly. In this case, frequentist alternatives may provide exactly that.

#Interpretation

“Interpretation is the wrong ground to concede.”

Michael Leznik, “Model-First, Not Bayes-First”

Michael makes two points about interpretation. First, the main promise of Bayesian output is not the probabilistic interpretation itself, but the ability to derive many other quantities from the posterior. Second, the quantities shown to experimenters do affect decisions, contrary to our claim.

On the first point, we agree that the posterior makes many useful quantities easy to derive. It can provide dispersion, threshold probabilities, conversion-versus-conditional-spend decomposition, and many other summaries of the fitted model. But it is not true that these quantities are unavailable outside a Bayesian framework. Frequentists can often use bootstrap for the same purpose. The main difference, as we see it, is the convenience of the mathematical machinery used to obtain them. That said, companies seeking Bayesian inference often use the probabilistic interpretation itself as an argument for adopting it.

Michael's second point is very interesting, and it suggests a stronger version of our criticism of the "more quantities are available" argument that we intend to adopt. We incorrectly stated that more information rarely changes the result. We should have said that it should not change the result if the decision procedure is to retain its intended properties. In a product organisation, programme goals must be achievable in practice across many teams. That means the quantities allowed to determine a decision must support the procedure chosen to deliver those goals. If you use Bayes-factor stopping to achieve a particular form of sequential error-rate control, but also show a conflicting posterior summary and let experimenters choose between them, you invite them to override the decision procedure intended to deliver that property.

More information is therefore not automatically a better decision system. Michael argues that the primary estimand should be pre-specified and the rest treated as descriptive. We agree, but labels alone are not enough if exploratory quantities routinely influence the primary decision.

In Confidence, we separate decision information from exploratory information so experimenters are not left to construct a new decision rule from whatever numbers happen to be visible. This is not because descriptive information lacks value. Interpretation, diagnosis, and learning serve different purposes from the programme-level decision. If conflicts between them are not resolved explicitly, but left to the experimenters with various intentions and statistical understanding, the decision rule becomes fragmented across individuals. For us, making sure experiment programmes deliver on their goals is central, and any fragmentation that undermines our ability to deliver on the goal is a problem.

#Model fit and validity

“The real division is not Bayesian against frequentist. It is model-first against test-first.”

Michael Leznik, “Model-First, Not Bayes-First”

Michael argues that test-first methods hide their modelling assumptions and can fail silently, whereas an explicit model can be checked using tools such as posterior predictive checks.

We agree that validity deserves more attention in the paper. A procedure is only useful if the assumptions needed for its intended property are plausible in the setting where it will be used.

At the same time, we argue that validity is not the same as model fit. Validity asks whether the procedure delivers the property it was chosen for under plausible data-generating processes. Model fit asks whether a probabilistic model reproduces relevant features of the observed data. Poor fit does not automatically invalidate a procedure, just as good fit does not guarantee the operating characteristics you care about.

Our analysis relies on the difference-in-means estimator being approximately normal, not on individual observations being Gaussian. Whether that approximation is good enough, especially for heavy-tailed metrics or sequential procedures, is itself part of the validity question. Posterior predictive checks can help diagnose potential model failures, but if such a check can change the decision, there also needs to be clear guidance about what happens when it triggers.

#What does robustness mean?

“The unasked question is whether error-rate guarantees are the only guarantees worth having.”

Michael Leznik, “Model-First, Not Bayes-First”

Michael argues that our comparison privileges programme-level error rates while giving too little attention to modelling choices, model checks, precision, and other properties an experiment might be designed to deliver.

To us, robustness only makes sense relative to a goal and a specified failure mode. A robust mixture prior may protect against prior-data conflict, but that protection can change other properties you care about. For example, say you are using Bayesian inference with an estimated empirical Bayes prior because you want shrinkage to counter the winner’s curse. An estimated treatment effect that is very large relative to the prior can mean two different things. It may reflect the winner’s curse, in which case the shrinkage is exactly what you wanted, or it may indicate that the prior is inappropriate for this experiment, in which case the robust component is useful. The data alone do not necessarily tell you which situation you are in. Robustification therefore trades one failure mode against another rather than simply making the procedure more robust. Likewise, BEST reduces the influence of extreme observations, which may be desirable for a robust notion of location but not necessarily when the target is the literal arithmetic mean.

The broader point is that choices about the estimand, the metric, and the observation model come before the comparison in our paper. We fix those choices and then compare configurations. Michael is right that we should make that boundary clearer.

Precision is also worth addressing directly. In a fixed-sample setting, MDE, power, alpha, and variance together determine sample size and therefore planned precision. You could instead start from the precision you want and let that determine sample size and the resulting power curve. These are different ways of specifying how informative the experiment should be. The broader point is that power is not the only relevant goal. If interval width, effect-size accuracy, or the ability to distinguish meaningful from negligible effects matters, that should be stated and designed for directly. We have written about precision based designs in the frequentist framework in this blog post and paper.

More generally, this reinforces our central point. Framework choice is not the first decision, and model choice is not the last. The important choices are the goal, the estimand, the assumptions needed to learn about it, and the rule for translating the resulting information into a decision. Any benefits of that rule must also be weighed against the operational complexity of implementing it across an experimentation programme. Once those choices and tradeoffs are explicit, many apparent Bayes-versus-frequentist disagreements turn out to be disagreements about one of them instead.

Thanks again, the paper will be better for this exchange.