To configure treatments for an experiment you first need to select the flag to use.
To select the flag, click + Add control at the top of the edit page.
After you have added a flag you can add treatments to the experiment. The first
variant you add is the control group. The next variants you add are treatment
groups. You can reorder the variants by dragging them left and right to select a
different control. You can read more about flags and variants on the
flags page.
Treatment Sizes
The treatment size is the size of each variant in the experiment. For example,
if you run an experiment with one control group and two treatment groups, you may split your
sample so that 40% of the users are randomly allocated to the control group, 40% to
the first treatment group, and the remaining 20% to the second treatment group.
Divide the sample evenly between the variants whenever possible. This gives you the highest power.
Do this by clicking SPLIT WEIGHTS EVENLY.
- You can run the test with an even split on a less risky part of the population
(for example, new users, free users or target a specific country)
- You can lower the allocation of the experiment to reduce the total population
you are targeting and have an even split that would expose less users. See
Audience for more information.
You need to be careful to ensure that the results are still generalizable. The statistical
tests let you conclude causally whether a change had an effect on
the metric that you observe for the population you are testing on. Testing on a
subpopulation could be a good first step to evaluate the risk before testing on
the rest of the population.
