Mutually Exclusive Experiments

Make experiments exclusive to other experiments by ensuring that they have at least one exclusivity group in common. For example, make two A/B tests exclusive to each other by adding the same exclusivity group to both A/B tests. No overlap means that no user or other entity is in both of these experiments at the same time. See the audience page on experiments for more information about exclusivity when running experiments. Using exclusivity groups, you can coordinate experiments that change the same or similar parts of an experience and make them exclusive to each other. For example, use exclusivity if you want to run multiple related experiments, but you want to only expose users to one of the experiments. If you don’t make the experiments exclusive to each other, all the multiple experiments can expose a given user. Watch this video to get a quick overview of how to use exclusivity groups to coordinate experiments in Confidence.

​

Exclusivity Groups

By using several exclusivity groups, you can create sophisticated coordination of experimentation programs. For example, a set of A/B tests can be exclusive to each other, but be randomly overlapping with another set of experiments, where the experiments in the second set are also exclusive to each other. Read more about exclusivity groups in the surface documentation.

​

How Exclusivity Works

If you want two experiments to be exclusive to each other, there must be enough users to allocate to both experiments. If there aren’t enough users to allocate, the experiments aren’t able to run at the same time. You define how much of the audience to allocate to an experiment by setting the allocation. For example, these two experiments can run at the same time, and be exclusive to each other:

• Experiment A: 50% allocation, exclusivity group A
• Experiment B: 50% allocation, exclusivity group A

These two experiments aren’t able to run at the same time since there aren’t enough users to allocate to both experiments:

• Experiment A: 50% allocation, exclusivity group A
• Experiment B: 70% allocation, exclusivity group A

Confidence takes inclusion criteria of the audience into account when allocating users to experiments. If the audience of experiment A is already exclusive to the audience of experiment B based on their inclusion criteria, then experiment A is able to run at the same time as experiment B even if their allocations are larger than 100%. For example:

• Experiment A: 50% allocation, exclusivity group A, inclusion criteria country = US
• Experiment B: 70% allocation, exclusivity group A, inclusion criteria country = SE

​

When Experiments should be Mutually Exclusive

In certain situations, being in multiple related experiments at the same time can lead to a poor and unexpected user experience because of dependencies among tests. Examples of when you should consider using exclusivity to avoid conflicts between experiments:

• If experiments are using the same flag, for example to alter the ranking of search results. Since only one experiment can decide the search rank for one user at one time, exclusive experiments allow several experiments on search rank simultaneously.
• If the variants of the experiments can interfere with each other. For example, if one experiment changes the background color of a page (using flag A), and another experiment changes the text color of the same page (using flag B).

In other cases it may not be as obvious if you should make the experiments exclusive to each other or not. A rule of thumb is to ask the question:

If these experiments run in sequence after each other, is it fine to ship the winning variants from each experiment without testing them together first?

If the answer to that question is “Yes” then it’s fine to let the experiments overlap and run them at the same time. If the answer is “No”, you should either test the changes in combination or run them separately and non-overlapping by making sure that the experiments are using the same exclusivity group.x If you test them separately, make sure that you then run a follow-up experiment where you test the promising cells from each experiment in combination with each other. What combinations to test depends on what treatment cells from the two experiments that can co-exist in a meaningful way. A rule of thumb is that you should run overlapping experiments when possible to save space for more simultaneous product evaluations.

​

Make an Experiment Exclusive to Other Experiments

To make an experiment exclusive to other experiments with a certain exclusivity group, follow these steps.

1. Go to Confidence and find the A/B test or rollout you want to make exclusive to other experiments.
2. Select the surface on which the exclusivity group lives.
3. Select the exclusivity group you want to use
4. Click save.

​

Coordinate Experiments Across Multiple Surfaces

Coordinate experiments across multiple surfaces by using exclusivity groups from more than one surface. Watch this video to build intuition for advanced experiment coordination in Confidence.

​

Related Resources