Lesson 9: Make a Decision
In this lesson, you learn about decision making in the context of experimentation.
To benefit from experimentation in your decision making you should:
- Have a pre-determined decision rule
- Ship successful variants, iterate on non-successful variants using explorations and experiments
Define a decision rule before you run the experiment
It is important to have pre-defined decision rule that maps any possible outcome of the experiment to a product decision. For example, what will you do if one guardrail metric moved in the wrong direction, but everything else looks good? Predetermining the rule makes it easier to not change the goal after you see the results.
In Confidence, there is a default decision rule that gives decision recommendations throughout the experiments. Read more about how the recommendations are constructed in the documentation.
Iterate on a product with experimentation
The following chart shows how to apply the scientific method when you make a decision based on an experiment.
If the experiment confirms the hypothesis and there is evidence that your change works well, then you can proceed and roll out the change to all users.
If the experiment is not successful, that is, if there is a negative effect detected on a guardrail metric or no success metric has improved, you should not roll out the change. Instead, you should try and understand why this iteration didn't have the intended effect, fix the problem, and then re-run the experiment to see if your new fix actually fixed the problem.
There are several ways of trying to understand why an iteration didn't have the intended effect. For example, you can do exploratory analysis by diving into segments and additional metrics to better understand your results. It is also common to do user research to get more in-depth understanding of how users experienced the change. All this information can then be used to formulate a new hypothesis, and iterate on the product. You then test the new iteration in a new experiment.
In Confidence, you can do exploratory analysis directly in the explore tab, diving into segments and additional metrics to better understand your results.
Although exploratory research and analysis is an important and natural step to inform new iterations, don't use it to make a decision on the finished experiment. Changing the prediction to match the results invalidates the conclusion.
Of course, it is not a good idea to iterate forever. If repeated experiments fail to show improvement, that is a signal to abandon the hypothesis rather than keep refining it.