Lesson 3: Means and relative effects
In this lesson, you learn what the control variant mean and treatment variant mean represent, why Confidence reports effects as relative percentages, and how to read the numbers shown for each metric.
For each metric in your experiment, Confidence shows you two numbers: the control variant mean and the treatment variant mean. From those, it calculates a relative change. Understanding what these numbers actually represent and why effects are shown as percentages makes everything else on the results page easier to read.
Control variant mean and treatment variant mean
An experiment splits users randomly into groups. The control group receives the current experience. Each treatment group receives a changed experience. After the experiment runs for a period of time, Confidence computes the average value of each metric for the users in each group.
- The control variant mean is the average metric value across all users assigned to the control variant.
- The treatment variant mean is the average metric value across all users assigned to a given treatment variant.
Both numbers reflect what actually happened for real users during the experiment. They are not predictions or model outputs; they are averages of observed data.
A metric tracks the number of items added to a cart per user. After two weeks, the control variant average is 2.45 items and the treatment variant average is 2.61 items. These are the control variant and treatment variant means shown in Confidence.
Why relative effects?
Rather than showing the raw difference between the treatment variant and control variant means, Confidence always shows the relative change: the difference expressed as a percentage of the control variant mean.
So in the example above: (2.61 − 2.45)/2.45 ≈ +6.5%.
There are two good reasons for this:
Comparability across metrics. Your experiment may include metrics measured in completely different units: seconds, click rates, counts, revenue. A 0.5-second improvement means something very different for a 5-second process than for a 5-minute one. Relative effects put everything on the same scale, making it much easier to compare results side by side.
Intuitive interpretation. A +5% improvement in conversion rate is immediately meaningful to most people, regardless of whether the baseline is 2% or 20%. The absolute difference is harder to assess without knowing the baseline.
Because the effect is expressed relative to the control variant mean, you always need the control variant mean to interpret the absolute magnitude of the change. A +5% change on a metric with a control variant mean of 0.01 is a much smaller absolute movement than a +5% change on a metric with a control variant mean of 1,000.
What to look for
When you read the metric results in Confidence, start with three things:
- The direction. Is the relative change positive or negative? Does that match what you expected from the treatment variant?
- The magnitude. Is the effect large enough to matter in practice? A +0.01% change on a revenue metric might be statistically detectable but commercially irrelevant.
- The confidence interval. This tells you the range of plausible values for the true effect. You will learn how to read it in the next lesson.
A note on adjusted means
When variance reduction is active for a metric (which it is by default in Confidence), the control variant and treatment variant means shown are slightly adjusted versions of the raw group averages. This adjustment is designed to produce a more precise estimate of the treatment variant effect. You will learn exactly how this works in Lesson 8. For now, the key point is that you should interpret the relative change shown in Confidence in the same way regardless of whether variance reduction is active.
The control variant mean for a metric is 100 and the treatment variant mean is 107. What is the relative effect shown in Confidence?
Why does Confidence report effects as relative percentages rather than absolute differences?
Notes for nerds
Even though Confidence displays effects on a relative (percentage) scale, all statistical testing is done on the absolute scale. The relative effect shown is simply the absolute estimated difference divided by the control variant mean, a transformation applied after the test is run.
This matters because it means Confidence does not need to account for the control mean being an estimate rather than a fixed number. The inference is based on the absolute difference, which has clean frequentist properties. If significance testing were done on the relative scale directly, the control mean would itself be stochastic across repeated experiments, complicating the statistical guarantees. By testing on the absolute scale and only transforming for display, Confidence avoids this issue entirely.