Lesson 2: True versus estimated effects
In an experiment, you don't observe the treatment effect in the full population. You only observe a random sample of the population in your experiment, and, the sample is randomly split into treatment and control.
In an experiment, you don't observe the treatment effect in the full population. You only observe a random sample of the population in your experiment with that sample randomly split into treatment and control. You only observe the mean of the non-treated group for the subset of users in the sample that are in your control group, and only observe the mean of the treated group for the subset of users in the sample that are in your treatment group.
Clearly, the estimated difference in means between the treatment and control group is not the same as the true treatment effect in the population. The estimated difference in means varies depending on which users end up in your random samples, and which of the users in the sample that end up in the treatment and control groups.
In the illustration below, three random samples are drawn from a population. The samples are split into treatment and control, and exposed to different variants of a mobile app. The samples are small, which makes the estimates very uncertain. In some samples, the estimated difference in means is larger than zero, in some smaller than zero. This variation is referred to as the sampling variation of the difference-in-means estimator: It's the variation of the difference-in-means estimator across random samples and treatment assignments.
A treatment effect estimator is said to be unbiased if the average of all estimates across all possible random samples and treatment assignments is equal to the true population treatment effect.
Separate the signal from the noise
So how do you know if the observed difference is due to random variation? Did users with a high value of the outcome metric by chance end up in the treatment group, or did the treatment actually have an effect? This is where statistics comes in.
Because the sample and treatment assignment is random, probability theory lets us quantify the uncertainty in the estimated difference in means under the null hypothesis. If the treatment has no effect, then any variation in difference-in-means estimates across random samples only occurs because different users with different outcome values happen to be placed in different treatment groups.
Probability theory lets us quantify how likely a certain mean difference is if the treatment has no effect (meaning that the null hypothesis is true). If the observed difference is very unlikely under the null hypothesis, we reject the null hypothesis and conclude that the treatment has an effect.
The idea of rejecting the null because the observed outcome is unlikely under the null can be challenging to digest. But this is important to understand to build intuition for experimentation.
We say that a mean difference is statistically significant if it's among the alpha percent most unlikely mean differences under the null hypothesis. If that's the case, we reject the null hypothesis and say that "we found evidence for the alternative hypothesis". Alpha is a parameter that the experimenter sets, we return to alpha in Lesson 5.
The logic of rejecting the null is that since it would be much more likely to observe a large mean difference if the treatment indeed had an effect, we rather believe that the treatment has an effect than believe that the null hypothesis is true and that we just observed a very unlikely mean difference by chance.
If this was your first time hearing about statistical significance and rejecting the null hypothesis, don't worry. This is a concept that takes time to understand. Go back to this lesson tomorrow or in a week an do it again. For most new experimenters, this takes a few attempts to understand.
But how can we know if an observed mean difference is among the alpha percent most unlikely mean differences under the null hypothesis? We can certainly not go through all samples and treatment assignments and give everyone no treatment (just in the example above there are more than 800 million combinations of samples and treatment assignments). In the next section, we dig into how we can know the distribution of the difference-in-means estimator under the null hypothesis without going through all possible samples and treatment assignments—using math.