Lesson 8: Set the sensitivity of the experiment with the minimum detectable effect (MDE)

Summary

In this lesson, you learn about the minimum detectable effect (MDE) and how you use it to set the sensitivity of an experiment.

The Minimum Detectable Effect (MDE):

  • Decides how small effects in your metrics you can detect.
  • The smaller MDE you choose, the larger the sample size you need for your experiment.

After deciding which metric to use to measure success, you need to define what effect size you want the experiment to be able to detect. This effect size is called the "minimum detectable effect" (MDE), or "minimum relevant effect". Use the MDE to set up the experiment so that it has enough sensitivity to detect meaningful effects.

Picking the MDE is a trade-off between:

  • the smallest effect relevant for the business
  • the smallest effect that's practically measurable with the sample size you can reach in your experiment

As an experimenter, you can use your domain expertise and discuss with stakeholders to decide what is the smallest effect that you would consider meaningful. In the next step, you calculate what sample size you need to be able to reliably measure this effect. If the sample size required to measure the chosen MDE is unrealistically large, then you need to adjust the MDE upwards.

One way to understand the minimum detectable effect (MDE) of an experiment is to imagine your experiment as a microscope.

Illustration: MDE as the resolution of a microscope

Microscope image

Imagine looking at tissue-sample under a microscope. The more you zoom in, the more details you can see. Changes to the sample that would be hard or impossible to see at one level of magnification become clear at a higher level of magnification. The minimum detectable effect (MDE) in an experiment is like the resolution of a microscope. It is the smallest change that you want to be able to see. If you want to be able to see smaller changes, you need a higher resolution. In experiments, you can increase the 'resolution' or sensitivity by increasing the sample size. The larger the sample, the smaller changes you can detect. Just like you cannot zoom in on a microscope indefinitely, you cannot detect arbitrarily small changes with an experiment, because you don't have an infinite number of users.

Learn about the Minimum Detectable Effect (MDE) and how to set it in your experiment in 3 minutes and 43 seconds.

Reader exercise

What is the purpose of the minimum detectable effect (MDE) in an experiment?