The Hidden Cost of Winsorizing

Most experimentation platforms handle outliers with a percentile threshold that moves with your data. That threshold is influenced by the very treatment effect you are trying to measure, which introduces a source of bias that is rarely discussed.

#One threshold is fixed. The other moves with your data.

There are two dominant methods for handling outliers in mean-based metrics.

Capping sets a fixed, pre-specified threshold based on domain knowledge. A $500/day revenue cap, for instance, because no legitimate transaction exceeds that amount. It does not change between arms, and it does not change across experiments.

Winsorizing sets the threshold at a percentile of the observed data, such as the 99th or 99.9th percentile. That threshold is computed from the data itself, which means it shifts depending on who is in the sample and how they behave.

In a descriptive analysis, where you are summarizing a single sample rather than estimating a causal effect, the distinction is minor. In an A/B test, it can affect decisions.

#Data-dependent thresholds let the treatment shift the boundary

In an experiment, the treatment changes behavior. If your treatment makes users spend more, the distribution of spending in the treatment arm shifts upward. When you compute a pooled percentile across both arms, the treatment arm contributes more high values, pulling the threshold up.

The clipping is asymmetric. Consider five control users who spend $8, $10, $12, $15, and $200, and five treated users who spend $10, $14, $16, $20, and $300. The raw means are $49 and $72, a treatment effect of $23. The pooled percentile threshold across all ten users falls at $200. After winsorizing, the $300 in the treatment arm becomes $200, the treatment mean drops to $52, and the measured effect shrinks to $3. More than 85% of the real effect has disappeared.

When two development economics experiments were reanalyzed using per-arm rather than pooled outlier procedures, treatment effect estimates changed by an average of 53%. In both cases the per-arm estimates were larger. A separate theoretical analysis shows that pooled winsorizing has a fundamental worst-case limitation: no single choice of percentile threshold works reliably across all distributions. Per-arm winsorizing does not have this limitation.

In the typical case where the treatment shifts the distribution upward, the result is attenuation bias. Pooled winsorizing makes a real treatment effect look smaller than it is.

#Per-arm winsorizing helps, but does not fully solve the problem

The natural response is to winsorize each arm separately. Compute the 99th percentile within the treatment group and within the control group, and clip each arm at its own threshold. This solves the pooled-threshold problem.

But it creates a subtler one. When each arm has its own threshold, the outcome transformation depends on treatment assignment. The estimand (the quantity your analysis is actually estimating) is no longer the same in both groups, because each group's outcome has been transformed by a different rule. You are comparing groups clipped at different thresholds, and the same study shows that the Type I error rate (the rate of false positives) can increase in finite samples. You reduce attenuation bias but inflate false positive rates.

Per-arm winsorizing works well when the treatment shifts the distribution without changing its shape. When the treatment also changes the shape or spread, the per-arm thresholds shift differently in each arm, which can introduce its own distortions.

#Most platforms default to percentile-based pooled winsorizing

As of mid-2026, most major experimentation platforms default to pooled percentile-based winsorizing. Statsig uses the 99.9th percentile pooled across arms. Datadog Experiments (formerly Eppo), LaunchDarkly, and Kameleoon all use percentile-based approaches, with LaunchDarkly and Kameleoon explicitly pooling across arms. Kameleoon pools across the entire account. GrowthBook and Statsig offer fixed-value capping as an alternative, but their percentile options are pooled across arms. Confidence supports only fixed-value capping, by design.

Most platforms do not surface the distinction between pooled and per-arm computation. If you cannot switch platforms, run sensitivity checks by comparing results at different percentile thresholds. If your conclusion changes when you move the threshold by half a percentile point, treat the result as fragile.

#Winsorizing does reduce variance

Metrics like revenue or session duration are heavy-tailed. A small fraction of extreme values drives most of the variance, and that variance can be enormous. Reducing that variance shrinks confidence intervals and lets experiments reach significance faster.

A paper on the tradeoff between bias and variance in experimentation argues that variance reduction can be worth some bias, particularly when you are making decisions across many experiments over time.

But you do not need a data-dependent threshold to get that variance reduction. A fixed cap based on domain knowledge also removes extreme observations, without letting the threshold move with the treatment. An earlier post on this blog recommends exactly this approach for reducing sample size requirements.

#Capping has a tradeoff too

Capping is not bias-free. If your treatment genuinely shifts values above the cap, capping also attenuates the treatment effect. A cap at $500 clips a user who moved from $400 in control to $600 in treatment, recording a $100 difference when the true effect was $200.

The difference is that you control this tradeoff. Set the cap too low and you attenuate real effects. Set it too high and you do not reduce enough variance. But you make that choice before seeing the results, based on what you know about the metric. With winsorizing, you have no control over that tradeoff. The threshold shifts after the treatment has already influenced the data.

This also means capping forces you to think about your metric. You need to decide what constitutes a reasonable maximum value and articulate why. For metrics with natural bounds, like the number of tracks played per day, this is straightforward. For unbounded metrics like revenue, it takes judgment about which values represent real behavior and which are logging errors or extreme edge cases.

That forced thinking is the point, and it surfaces an important distinction. A $50,000 daily revenue value is almost certainly a logging error. That is a data quality problem: the observation should be excluded or corrected by a predefined validation rule, not handled by a statistical transformation. A $500 daily revenue value when the median is $10 is probably a legitimate power user. That is a statistical problem: the observation is valid but extreme, and you are making a deliberate bias-variance tradeoff by capping it. A fixed cap forces you to think about which problem you are solving. A percentile threshold treats both cases identically. The same logic applies to lower bounds. Net revenue after refunds, for example, can have extreme negative values that deserve a floor cap for the same reasons.

Winsorizing skips that step. You pick a percentile, apply it, and move on. The ease is part of its appeal, and part of the problem. You never have to engage with what the numbers mean, and you are left with a bias you did not reason about.

This is why Confidence only supports capping. When you genuinely cannot set a fixed cap (some marketplace or ad-revenue metrics have no natural ceiling), per-arm winsorizing is preferable to pooled winsorizing. Use a high percentile and verify that your results are not sensitive to the threshold you chose. But start by asking whether a fixed cap is possible. For most metrics, it is.

There is also a modeling question worth raising. Rather than post-processing data to handle extreme values, you could use a model that accommodates heavy tails directly, such as a log-normal or robust regression. That approach avoids the bias-variance tradeoff of capping and winsorizing entirely, at the cost of stronger distributional assumptions.