Lesson 4: Experiment hypothesis
In this lesson, you learn how to create a plan for your experiment. You learn how to formulate a hypothesis that acts as the product foundation for your experiment, and get examples of essential questions to ask yourself when planning your experiment so that you don't run into problems later on.
A good hypothesis:
- Is short and direct.
- Is testable in the sense that the experiment can give clear evidence for or against it.
Define your hypothesis
Before you run an experiment, you need to formulate a hypothesis statement. Use it to articulate what you plan to test, and how. When you run an experiment, you actually do hypothesis testing, so this step is important!
The process of formulating a hypothesis allows (or forces) you to think through the basis for what you are testing, and put this into writing. A well formulated hypothesis should contain:
- What prior information led to this hypothesis.
- What change you make.
- For whom (typically which users) you make the change.
- What you hope the change achieves.
- How you plan to decide whether it was successful.
Hypothesis template
A template that you can use to formulate this is:
Based on [prior knowledge], we believe that [theory about user need]. We think that [doing this/building this feature/creating this experience] for [these people/personas] will achieve [these outcomes]. We will know this is true when we see [metric results].
Using a change in the sign-up flow as an example, you could formulate a hypothesis as follows:
Based on user research, we believe that having to create a username creates friction in the signup process. We think that removing the step to enter a username for users signing up in the app will lead to more users successfully completing the signup flow. We will know this is true when we see an increase in the sign-up completion rate.
A strong hypothesis should also describe why you believe this change will achieve the desired outcome. You should back it up by earlier research, data, or domain knowledge (and not just base it on a hunch).
*Go from Goal to Hypothesis, adapted from the Thoughtful Execution framework.
In a later lesson, you will learn more on how to define success, including how to select success metrics, and how to think about at what point you will consider a change in a metric to be a sign of success. For example, by how much does the sign-up completion rate need to increase for you to consider it a success—by 1%? Or as little as 0.1%? More about this later!
Ideas, that could become fully defined hypotheses, can come from anywhere—an engineer, a designer, customer support, or an end-user of your product. Many ideas could result in product changes and new features. Without testing them, you won't actually know if you were correct and that the change in fact made the product better. Experiments help you do that!
Make sure you're good to go
When the hypothesis is starting to take shape, it's time to also consider things like:
- How do you plan to build the experience that you want to test? Who do you need to involve to make it happen?
- Do you need to sync with any other teams about what you are doing? For example, are you using, modifying or impacting part of your product that another team owns?
- Do you need to coordinate your experiment with any current or future other activities?
Doing this kind of thinking and planning early on can save you a lot of time and effort later on!
Are the metrics you want to track already available in Confidence, or do you need to set them up?