# Confidence Bootcamp — Full Content
> This is the complete text content of the Confidence Bootcamp, a free learning program covering experimentation, metrics, hypothesis testing, sample size calculation, and experimentation culture.
>
> Source: https://confidence.spotify.com/bootcamp
---
## Table of Contents
- [Intro to experimentation](#intro-course)
- [Interpreting experiment results](#interpreting-experiment-results)
- [Intro to metrics](#intro-to-metrics)
- [Scientific product development](#scientific-product-development)
- [A primer on hypothesis testing](#hypothesis-testing)
- [Sample size calculation I](#sample-size-calculation-i)
- [Sample size calculation II](#sample-size-calculation-ii)
- [Sample size calculation III](#sample-size-calculation-iii)
- [Advancing your experimentation](#advancing-experimentation)
- [Experimentation culture](#experimentation-culture)
- [Intro to feature flags](#intro-to-feature-flags)
- [Videos](#videos)
---
## Intro to experimentation
Source: https://confidence.spotify.com/bootcamp/intro-course/introduction
# Welcome to *Intro to experimentation*
[Interactive: CourseProgressBar]
*Intro to experimentation* is an asynchronous, self-paced course where you dive
deeper into experimentation. While experimentation is a broad
topic, this course focuses on teaching you the basics of A/B testing. Much of
what you learn in the course also applies to other types of experiments, like
rollouts.
In the course, you learn the key concepts of A/B testing and how
to set up, run, and monitor an A/B test—interchangeably referred to as an experiment—using Confidence. The focus of the course is on
A/B tests, but much of what you learn in the course also applies to rollouts and
other types of experiments.
Before you start this course, you should have a basic understanding of what it means to run an experiment end-to-end. In our experience, this helps you internalize the concepts in our courses.
**In Confidence:** If you haven't already, go through the [A/B test quickstart](../../docs/abtests/quickstart) to get hands-on experience running an experiment in Confidence.
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material.
## Lessons
This course consists of the following lessons:
- Lesson 1: Why you should experiment: Learn about the benefits of experimentation and how it can help you make better decisions.
- Lesson 2: Experiment hypothesis: Learn how to plan your experiment and craft a solid product hypothesis.
- Lesson 3: Success and guardrail metrics: Learn about the roles of success and guardrail metrics in an experiment.
- Lesson 4: Success metrics: Learn how to select metrics and how to configure the sensitivity to detect effects.
- Lesson 5: Set up your experiment: Learn how to set up an experiment, and how to collaborate with your team in Confidence.
- Lesson 6: Calculation frequency: Learn about the options for when to calculate results and how to choose between them.
- Lesson 7: Target audience: Learn how to select who to target with your experiment.
- Lesson 8: Sample size: Learn how to calculate the sample size you need to detect the effects you are interested in.
- Lesson 9: Quality assurance: Learn about the steps you can take to ensure that your experiment works as intended before you launch.
- Lesson 10: Run your experiment: Learn about what to keep track of while your experiment is live.
- Lesson 11: Evaluate your experiment and make a decision: Learn how to interpret the results of your experiment and decide the next steps for the product you are iterating on.
- Lesson 12: Choosing between A/B tests and rollouts: Learn how to choose between A/B tests and rollouts and when to use both.
---
Source: https://confidence.spotify.com/bootcamp/intro-course/ab-tests-and-rollouts
# Lesson 12: A/B tests and rollouts
> Use A/B tests to identify the winning variant, use rollouts to ship the
winner. For technical changes, like major refactors and migrations, use
rollouts to avoid risk and to be able to roll back instantly.
A/B tests and rollouts are two tools for product evaluation. Although similar in some ways, they are usually used for different stages of evaluation.
## A/B tests
The main characteristics of A/B tests are that they
- Can have more than two variants
- Can have both success metrics and guardrail metrics
- Have a fixed allocation of the total population
- Allow for different evaluation frequencies for calculating results
**Use A/B tests to**
- Decide a winner among two or more variants of a product
- Explore and learn about how different settings affect use behavior
A/B tests are flexible and rich product evaluation tools. They help you ensure that the winning variant is better than the losing variants
for the business, by allowing you to consider a complete set of success and guardrail metrics.
## Rollouts
The main characteristics of a rollout is that it
- Can only have two variants of which one is the current default and one is the variant that you want to roll out
- Can only have guardrail metrics
- Has an allocation that you can gradually increase
- Always displays results continuously
**Use rollouts to**
- Gradually ship a variant while monitoring important guardrail metrics
- Gradually ship technical changes to the system, for example major refactors and migrations
A great benefit of using rollouts to ship changes is that the change that
you roll out is behind a feature flag. This makes it easy to roll
back a change. In other words, if you start rolling something out and get an
alert that the rollout harms the end-user experience, it is only a button click
away to revert back to the earlier experience. This saves engineers a lot of
time and agony at Spotify, and has made rollouts the default way for
engineers to release changes.
## Combine A/B tests and rollouts for important changes
Significant results from experiments with small sample sizes tend to over
estimate the treatment effect. Some online experimenters propose that you should
replicate the results from such experiments by rerunning the experiment on other
users to confirm the result. In practice, it
is hard to know which experiments are underpowered. One way to think about it is
that you should scrutinize unexpected results harder, and replicate them, to
believe in them. A practical way to get confirmation of results from an A/B test
is to ship the winner with a rollout. The metric results in the rollout works as
a replication of the A/B test results and you can be even more certain about
making the right decision.
At Spotify, most A/B tests that identify a winning variant use a
rollout to ship that variant, which means that most results are
replicated while released to everyone.
**In Confidence:** In Confidence, A/B tests and rollouts are the two main experiment types. Rollouts are behind feature flags, making it easy to revert with a single click if something goes wrong. Read more about [A/B tests](../../docs/abtests/quickstart) and [rollouts](../../docs/rollouts/quickstart) in the documentation.
**Quiz: You should choose a rollout instead of an A/B test when you want to:**
- Find a winner among two or more variants of a product.
- Gradually ship technical changes to the system, for example major refactorings and migrations. (correct)
- Explore and learn about how different settings affect use behavior.
---
Source: https://confidence.spotify.com/bootcamp/intro-course/calculation-frequency
# Lesson 6: Calculation frequency
> * Calculating results only once ("Upon Conclusion") is the most efficient way
to set up your experiments. You only get to see the results at the end of your
experiment, which gives you higher sensitivity.
* Calculating results continuously sacrifices sensitivity for faster results. Select this if getting
a rough estimate early on is important to you.
When you set up an experiment, you need to decide when you want results to be
displayed.
## Deliver results continuously or upon conclusion
There are two options for when to calculate results:
- **Continuously** display results to see new results every hour or day when the experiment is live.
- **Upon Conclusion** to see results when you end the experiment.
For both settings, all experiments run until you decide to stop them. The largest difference between
the two options is that in experiments with results delivered **Upon Conclusion**, you can't view
the results for your metrics *during* the experiment. The results show up when you end your
experiment. For experiments using results calculated *Continuously*, you can follow the experiment
results during the course of the experiment and decide to end it whenever you want.
If you're wondering why not always choose to view results continuously—stay tuned! We'll explain
that shortly.
## Two strategies to avoid being fooled by randomness
Imagine that you run an experiment that truly has no impact whatsoever on any metric. You just split
users in two groups, measure some outcome, and calculate the difference between the groups every
day. Even if the experiment didn't actually change anything, you can expect to see results fluctuate
over time, just by random chance. If you check the results every day, you might on some days see a
difference that's so large that you wrongly assume that this experiment impacts the metric. Such
results are called false positive results. Checking the results every day means there are multiple
chances to find a false significant result. To avoid getting fooled by randomness and draw the wrong
conclusions based on a false positive result, you can use either of two strategies to keep this risk
under control.
- ### Calculate result upon conclusion and use standard statistical tests
Before starting your experiment, you define a point in time when you plan to calculate the
results. To make sure that you have a good chance of finding an impact, you calculate what sample
size you need to reliably detect an effect of a certain size. By only calculating the results this
one time, you have only one chance to be fooled by a false positive result. This method is the
most efficient way to minimize the risks of false positives.
- ### Calculate the results continuously and use sequential tests
You calculate results every day or every hour, and correct for the increased risk of being fooled
by randomness. The statistical methods used, known as sequential tests, correct for the multiple
peeks at the data by using a stricter threshold to conclude significance. With this approach, you
can check the results daily without hesitation. Because the tests use a stricter threshold for
calling significance, the impact needs to be larger to be reliably detected. This means that for
an experiment with results updated continuously, you either need to increase the sample size, or
accept that you lose some sensitivity to detect changes. Sequential tests trade off sensitivity
for faster results.
**Learn about difference between different evaluation frequencies in 2 minutes and 14 seconds.**
[Video: https://www.youtube.com/watch?v=bMmlPKTt4V8]
## Automatic sequential monitoring
Modern experimentation platforms run automatic sequential monitoring checks regardless of the evaluation frequency you choose. This means that **you do *not* have to choose to calculate results continuously to sleep well at night**—the platform monitors your experiment for deterioration in all your metrics.
**In Confidence:** Confidence [monitors](../../docs/monitoring) your experiment using sequential tests for all checks, regardless of the evaluation frequency you choose.
## What you should choose
The trade-off is different in all experiments. Sequential experiments offer you the opportunity to
stop experiments early at the expense of some certainty. If speed is more important than precise
estimates of the treatment effect, continuously updating the results is the better choice.
For a given experiment length, like say two weeks, only calculating the results once at the end
gives more precise estimates. If you are going to run the experiment for a fixed amount of time
regardless, select **Upon Conclusion** to maximize your chances of finding effects.
**Example**
At Spotify, most experiments use *Upon Conclusion* and only display results
when the experiment ends. For early abortion of experiments due to errors or
negative user experiences, experimenters rely on Confidence's monitoring of
their experiments. For the product decision, most teams want at least two
weeks of data to make decisions to ship.
**Quiz: What setup should you select if you need to be able to see results every day?**
- Upon Conclusion
- Continuous (correct)
- Upon Conclusion or Continuous, both give results every day
---
Source: https://confidence.spotify.com/bootcamp/intro-course/evaluate-your-experiment-and-make-a-decision
# Lesson 11: Evaluate your experiment and make a decision
> In this lesson, you learn how to interpret the results of your experiment and
make a decision based on the results and how Confidence helps you do that. Use
exploration to learn more about the results you got and to get
inspiration for new hypotheses.
A good experimentation platform calculates results for you and displays the performance of each variant, taking care of the statistical details so you can focus on learning from the experiment. With those insights, you make a decision on how to proceed with the change you tested.
At this stage, your experiment has successfully run for a period of time and has no visible errors.
Congratulations! Now it's time for the fun part. You have at least one result to interpret, but
often there are more than just one. More precisely, your experiment has `T x M` results to
interpret, where `T = Number of treatment groups (excluding control)` and `M = Number of metrics`.
For an experiment with 3 treatment groups and 4 metrics, you have 12 results to interpret.
## Overall decision recommendations
A good experimentation platform provides overall decision recommendations that use the outcomes of all metrics to suggest whether a specific treatment is worth rolling out.
The shipping recommendation recommends you to ship a change if at least one success metric has moved
in the desired direction with significance. Simultaneously, all guardrail metrics must be
significantly non-inferior, meaning that they're all within the acceptable margin you set using the
non-inferiority margin. The test must also be in a healthy state, with no significant negative
changes in any of the metrics, and no sign that there is a problem with the quality of
the test.
**In Confidence:** Confidence provides [overall decision recommendations](../../docs/analyze-an-experiment#spotlight) on each treatment card on the results page.
## Metric results
For each metric, you see a comparison between the control group and each treatment group. You can
dig deeper into the results to see metric values, confidence intervals, variances, and more. If you ran your experiment with results delivered continuously, you can also view the results over time.
## Exploration
If at the end of the experiment you find things that you would like to dig deeper into, you can do exploratory analysis. Here you can add any metric and see how it performed for each of the treatment groups, and split the results by dimensions.
**Note:** This type of explorations in which you look at many metrics, perhaps until you find an
"interesting" result, **severely increases the risk for finding false positives**. This means you
risk that results are significant only by chance.
For that reason, you shouldn't use exploratory analysis to make decisions
about whether an experiment was successful or not. Use it to get inspiration for new hypotheses.
**In Confidence:** Use the [Explore tab](../../docs/exploration) to add any metric and split results by dimensions.
**Quiz: After ending your experiment, the results page in Confidence tells you that 'Shipping might be recommended' for the treatment. What does this mean?**
- Some success metrics have moved significantly in the desired direction, but some guardrail metrics are not significantly non-inferior (correct)
- Confidence doesn't know what your definition of success is, and therefore can't make a recommendation
- Some metric has deteriorated significantly
**Quiz: You ran an A/B-test and it turned out to show no significant difference between the treatment and control. However, you then go into the Exploratory tab, and after looking at about 10 different metrics, you find that there is a significant difference for one of the metrics for the treatment group. What do you do?**
- Declare that the treatment is a winner and send out emails to stakeholders that the experiment was a success
- Consider the high possibility that the significant result is a false positive, and use the result to form a new hypothesis and replicate this finding to see if it holds (correct)
- Only one significant result is a bit thin, so you look for more significant results in additional metrics
---
Source: https://confidence.spotify.com/bootcamp/intro-course/experiment-hypothesis
# Lesson 2: Experiment hypothesis
> In this lesson, you learn how to create a plan for your experiment. You learn
how to formulate a hypothesis that acts as the product foundation for your
experiment, and get examples of essential questions to ask yourself when
planning your experiment so that you don't run into problems later on.
## Define your hypothesis
Before you run an experiment, you need to formulate a *hypothesis* statement. Use it to articulate
what you plan to test, and how. When you run an experiment, you actually do hypothesis testing, so
this step is important!
The process of formulating a hypothesis allows (or forces) you to think through the basis for what
you are testing, and put this into writing. A well formulated hypothesis should contain:
- What prior information led to this hypothesis.
- What change you make.
- For whom (typically which users) you make the change.
- What you hope the change achieves.
- How you plan to decide whether it was successful.
### Hypothesis template
A template that you can use to formulate this is:
**Based on** [prior knowledge], **we believe that** [theory about user need]**. We think that**
[doing this/building this feature/creating this experience] **for** [these people/personas] **will
achieve** [these outcomes]**. We will know this is true when we see** [metric results].
**Note:** Using a change in the sign-up flow as an example, you could formulate a hypothesis as follows:
**Based on** *user research,* **we believe that** *having to create a username creates friction in the signup
process.* **We think that** *removing the step to enter a username* **for** *users signing up in
the app* **will lead to** *more users successfully completing the signup flow.* **We will know
this is true when we see** *an increase in the sign-up completion rate*.
A strong hypothesis should also describe *why* you believe this change will achieve the desired
outcome. You should back it up by earlier research, data, or domain knowledge (and not just
base it on a hunch).
\*Go from Goal to Hypothesis, adapted from the Thoughtful Execution framework.
**Note:** In a later lesson, you will learn more on how to define success, including how
to select success metrics, and how to think about **at what point you will
consider a change in a metric to be a sign of success**. For example, by how
much does the sign-up completion rate need to increase for you to consider it
a success—by 1%? Or as little as 0.1%? More about this later!
Ideas, that could become fully defined hypotheses, can come from anywhere—an engineer, a designer,
customer support, or an end-user of your product. Many ideas could result in product changes and new
features. Without testing them, you won't actually know if you were correct and that the change
in fact made the product better. Experiments help you do that!
## Make sure you're good to go
When the hypothesis is starting to take shape, it's time to also consider things like:
- How do you plan to build the experience that you want to test? Who do you need to involve to make it happen?
- Do you need to sync with any other teams about what you are doing? For example, are you using,
modifying or impacting part of your product that another team owns?
- Do you need to coordinate your experiment with any current or future other activities?
Doing this kind of thinking and planning early on can save you a lot of time and effort later on!
**In Confidence:** Are the metrics you want to track already available in Confidence, or do you need to [set them up](../../docs/metrics/quickstart)?
**Quiz: Which of the following do you consider to be the most complete and testable experiment hypothesis?**
- We believe that a simpler signup flow will lead to more users signing up to Spotify.
- We believe that a simpler signup flow will lead to more users signing up to Spotify. We will know this is true when we see an increase in the sign-up completion rate.
- We believe that removing the step to enter a username for users signing up in the iOS app will lead to more users successfully completing the signup flow. We will know this is true when we see an increase in the sign-up completion rate. (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-course/quality-assurance
# Lesson 9: Quality Assurance
> To test whether your code works as intended, use override rules to assign
specific users to your new feature. You can also run experiments on employees
only, and run A/A tests to test your setup end-to-end before launching your
main experiment.
## Overrides
You can assign a specific user to a particular treatment by overriding the randomization.
This means that you can add yourself or other members of the experimenting team to a specific variant at any time to try it out. You can verify that your implementations appear to be working as they should before releasing the experiment to actual users.
Overriding users into specific treatments doesn't affect the results, as the exposure data doesn't include the overrides.
**In Confidence:** In Confidence, you create [override rules](../../docs/define-rules#override-rules) to assign specific users to a particular treatment.
## Employee only
Depending on the nature of your product, a powerful next step in the QA process is to run your
experiment on employees only. Make sure to include an attribute in the evaluation context that
identifies the incoming request as belonging to an employee, and then use that in your inclusion
criteria. This way, you can test your change and its different values on users that are a bit more
forgiving. It can give you the chance to detect errors that you might not notice during
the early stages of QA. The drawback is of course that the sample size is typically so small that
it's difficult to find any meaningful effects, but you might hear from your colleagues if something
isn't working as it should.
**Note:** For employee experiments to be possible you must include employee status in
the evaluation context of your feature flag.
You can also give your new feature to employees only by directly creating a rule on your flag that
has employee status as an inclusion criteria.
## A/A tests
Sometimes you may want to run an A/A test to test your overall setup before launching the actual
experiment. An A/A test is
just like an A/B test, except that the experiences given to the control and treatment groups are the
same. Either the two variants you use are the same, or you resolve the flag in your code but don't
use the received variant values. A/A tests are particularly helpful if you want to test your whole setup end-to-end on
real users and get real exposure data.
**In Confidence:** A/A tests are useful when you have just integrated your service with Confidence. The [A/B test quickstart](../../docs/abtests/quickstart) describes such a test.
**Quiz: Whose responsibility it is to ensure that an experiment/rollout doesn't break the end-user experiment?**
- Statisticians and all other applied mathematicians
- The experimenters, that is, all of us who are running experiments (correct)
- The experimentation tool is responsible for everything
---
Source: https://confidence.spotify.com/bootcamp/intro-course/run-your-experiment
# Lesson 10: Run Your Experiment
> In this lesson, you learn how to monitor your experiment after launch, including checking exposure counts, reviewing automatic monitoring checks, and accessing your results either continuously or once the experiment ends.
After you launch your experiment, you should monitor that everything works fine. A good experimentation platform supports you in this stage with automatic monitoring checks. In this section of the course you learn what those checks are, and how to monitor that your experiment delivers your new experience.
## Directly after launch
When you launch the A/B test, exposure calculations run at frequent intervals initially. This makes it possible to directly see that the experiment works as intended.
**In Confidence:** Confidence runs [automatic monitoring checks](../../docs/monitoring) throughout the experiment. Hover the `Live` status on the right sidebar to see their current status.
If you use Confidence's flags, you can also see the real-time resolve count alongside the [exposure schedule](../../docs/exposure#schedules).
## At any time while the experiment is live
As soon as the first exposure and metric calculation finish, you can check the status of monitoring checks for your experiment.
**If you configured your experiment to deliver results continuously,** the full results page is
available while the experiment is live with results for all metrics that have at least some data.
## When you end the experiment
**If you configured your experiment to deliver results at the end of the experiment,** the results
page only shows the results for all metrics after you end it.
You can also use exploratory analysis to further drill down the results of metrics by splitting the results by dimensions, and add new metrics that weren't part of
the initial experiment setup. You can, for example, split the results by market to see how results
differ between different markets, or add a metric that measures something similar to an existing
metric to make sure that an unexpected result is present also in the new metric.
You can use exploratory analysis during the experiment, but to prevent p-hacking and cherry picking of results, you should wait until the experiment has ended before exploring additional metrics.
**In Confidence:** Read more about how to [interpret the results](../../docs/analyze-an-experiment) in Confidence. Use the [Explore tab](../../docs/exploration) to split results by dimensions and add metrics beyond the initial setup.
**Quiz: Which is a correct statement about monitoring and validation checks?**
- You have to use continuous results calculations to activate monitoring checks
- Only rollouts have monitoring checks
- Confidence alerts you if any validation check is failing for any of your experiments (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-course/sample-size
# Lesson 8: Sample size
> Sample size calculations help you understand the number of users you need to
include in your experiment to detect the effect of your new feature. This
lesson explains how to calculate the sample size for your experiment and
interpret the results.
## Sample size for experiments
**Sample size** refers to the number of units (most often users) included in an experiment. A larger
sample size increases the sensitivity of the experiment, and allows detecting smaller
differences between treatment and control group. Each metric has a hypothetical effect size known
as the minimum detectable effect (MDE) for success metrics, and the non-inferiority margin (NIM) for
guardrail metrics.
If you want to master sample size calculations, take the
[Sample size calculation - level I course](../sample-size-calculation-i/introduction).
Learn about the MDE and NIM in these short videos:
A smaller MDE or NIM requires a larger sample size. If you add more metrics, the required sample
size increases. The variance of each metric impacts the required sample size. Metrics that measure a
*quantity per user* (for example *Minutes played per user*) usually require a larger sample size
than metrics that measure the *Share of users who [completed some action]*.
The sample size calculation tells you the sample size that your experiment needs. This is often called the **required sample size**. If the sample size you expect to reach is larger than the required sample size, you can be confident that you have collected enough user data to see if your new feature had the intended effect.
**In Confidence:** Confidence calculates the required sample size based on your experiment setup. Click the **Calculate** icon on the right sidebar once you've configured your metrics and MDE. The calculation uses historical data and doesn't start your experiment or expose any users.
If the required sample size turns out to be unrealistically large (for example, twice as much as you
expect to reach), you need to go back and edit the settings of your experiment. For example, select
fewer metrics or adjust your expectations for what you can reliably detect and aim for a larger MDE
or NIM.
The results show the total required sample size alongside a per-metric breakdown. The metric with the highest required sample size determines the total, making it easy to spot the bottleneck and decide if you need to drop or adjust a metric.
**In Confidence:** 
In this example, the crash rate metric has a much higher required sample size than the checkout metric. Increasing its MDE or NIM would bring the total down. Learn more about [adjusting the required sample size](../../docs/results-settings#adjust-the-required-sample-size).
**Quiz: Which of the following tasks is NOT a fitting one for the sample size calculator?**
- Calculate the probability that I will find a significant result in my experiment when I run it (correct)
- Check if my experiment setup is reasonable for the sample size I need to be confident about the results
- Help me evaluate how long I need to run my test
---
Source: https://confidence.spotify.com/bootcamp/intro-course/set-up-your-experiment
# Lesson 5: Experiment setup
> This lesson goes over the main steps of setting up an experiment.
After the planning stage, you need to translate the details of your plan into an actual **experiment
configuration**. At this stage you create an experiment and configure it
according to your plan, which includes:
- Configure the test variants: What part of the user experience will the test change for the users?
- Select your target audience: For which users will the experience change?
- Select the metrics you want to track: What are the key success metrics you want to measure? What are important guardrail metrics?
## Configure the test variants
To set up an experiment, you need to define the test variants and connect the variants to the product code.
Feature flags are the standard mechanism for connecting experiment variants to your product.
Watch this video to get a quick understanding of how feature flags let Confidence control parts of your product remotely.
**In Confidence:** To get started with feature flags in Confidence, follow the [feature flag quickstart](../../docs/flags/quickstart). If you want to go through the full steps of setting up an experiment, follow the [A/B test](../../docs/abtests/quickstart) or [Rollout](../../docs/rollouts/quickstart) quickstarts. These guides have onboarded thousands of experimenters at Spotify and are a great way to get started.
## Select your target audience
When setting up an experiment, you need to define the target audience for the
experiment. This is the group of users that will be included in the experiment.
You can define the target audience based on different user attributes, such as
country, platform, or user age. What you can target on is determined by what
information is passed in when you resolve the feature flag.
Watch this video to get a quick understanding of how targeting and feature flag resolving are related.
## Select metrics
Refer to
[Lesson 3](./success-and-guardrail-metrics)
and
[Lesson 4](./success-metrics)
for details on the types of metrics and how to select them.
## Sample size and design
One important aspect of the experiment setup is the required sample size. The
required sample size is the number of users that you need in your experiment to
reach a level of precision in your results that lets you answer the question
you have set out to answer. You can control the sample size by setting the
allocation, the proportion of the target population that your experiment will
include.
Learn more about sample size calculations in [Lesson 8.](./sample-size)
**Quiz: What does the allocation control?**
- The proportion of experiments with a false result.
- The proportion of the target population that your experiment will include. (correct)
- The proportion of the target population that will not be in your experiment.
---
Source: https://confidence.spotify.com/bootcamp/intro-course/success-and-guardrail-metrics
# Lesson 3: How to measure impact with success and guardrail metrics
> Use success metrics to capture what you want to improve. Use guardrail metrics
to capture what you don't want to affect negatively.
When you run an experiment, such as an A/B test or a rollout, the ultimate goal
is to learn about the impact of the change you made. To know what the impact is,
you need to measure the outcome on a relevant set of metrics. The metrics you
select can serve different purposes, and even be subject to different
statistical tests. This page describes the two main types of metrics you can use
to measure impact, and how to select them.
The two types of metrics are:
- **Success metrics**. Metrics that you aim to improve with your change.
- **Guardrail metrics**. Metrics that you don't expect to improve, but that you want to make sure you
don't have a negative impact on.
## Success and guardrail metrics
**Success metrics** are the metrics that you aim to improve with your change. They're what you use
to prove that your change had a positive impact.
In companion to success metrics, you should also select **guardrail metrics.** Guardrail metrics are
metrics that help you make sure that your change doesn't have a negative impact on other aspects
of your product. This means a hypothesis for an experiment includes two criteria: one for the success
metric and one for the guardrail metric. Both criteria need evidence to support the decision to
launch the change.
Let's look at some examples of success and guardrail metrics.
### Example: Checkout flow
You run an A/B test with an improvement to the checkout flow of your e-commerce
website. Your goal is to make the checkout flow more efficient so that your
visitors spend less time in the checkout flow. You want to make sure that the
improvement in the checkout flow doesn't come at the expense of the number of
purchases.
- Success metric: Average time to completed checkout per visitor.
- Guardrail metric: Number of purchases per visitor.
### Example: Search algorithm
With your new Spotify search algorithm, your hypothesis is that users get better
podcast recommendations. You want to measure the impact of the new algorithm on
the consumption of podcasts. You want to make sure that the new algorithm change
doesn't reduce the number of users that listen to music.
- Success metric: The average
number of podcast minutes played per user.
- Guardrail metric : The average number of
music minutes played per user.
### Example: Dating app
You have a dating app that requires new users to complete their profile before
they can interact with others. You run a test where your hypothesis is that,
showing a dialog with advice for how to onboard, increases the number of users
that complete their setup. The dialog shown in your dating app experiment uses
new technology, and you want to make sure you don't introduce any bugs.
- Success metric: Share of users that complete their profile setup.
- Guardrail metric: Number of crashes per user.
**Recommendation:** When you start out with experimentation, it is a good idea just to select some guardrail metrics and start experimenting. After you got the hang of it, you can
make guardrail metrics even more valuable to your decision making by specifying so-called non-inferior margins (NIM).
Learn more about the tests used for guardrail metrics in
[this Advance your experimentation course lesson.](../advancing-experimentation/guardrail-metrics-and-nims)
**Example**
You are trying to increase the engagement in your product with a new variant
that is aiming to increase the engagement in a certain view, say A. To ensure
that an increase in engagement in view A doesn't come at the expense of
engagement in a competing view B. You should use the engagement in view A as
the success metric and the engagement in view B as the guardrail metric. If
the new variant increases the engagement in view A and doesn't decrease the
engagement in view B, the variant is successful and should be shipped.
## How success and guardrail metrics together define a successful variant
For a change to be worth shipping, at least one success metric must have improved significantly, while all guardrail metrics must not have regressed beyond acceptable limits. Both conditions must hold: a win on the success metric doesn't override a failure on a guardrail.
**In Confidence:** Confidence uses both success and guardrail metrics to identify a successful variant in A/B tests. Read more
about how success and guardrail metrics feed into
[the overall recommendation for a decision](../../docs/analyze-an-experiment#spotlight).
**Quiz: What is a guardrail metric used for?**
- It is used to guard that the success metric is significantly superior.
- It is used to prove that a metric has improved significantly as a consequence of the treatment.
- Guardrail metrics are used to find evidence for that a metric has not been negatively impacted by the treatment more than a certain amount. (correct)
**Quiz: What is a correct statement?**
- Choose guardrail metrics or success metrics, never both.
- Success metrics are better than guardrail metrics for most experiments. But in some cases, guardrail metrics are more important.
- Success metrics and guardrail metrics capture different aspects of overall success, and usually both should be used. (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-course/success-metrics
# Lesson 4: Success metrics
> In this continuation of lesson 3, you learn how to define success metrics for
an experiment, which is what you use to evaluate whether your change is
successful. You also learn how to reason about the sensitivity of the
experiment by defining the minimum detectable effect (MDE).
After you've written the hypothesis, you should have a clear idea which user behavior the
experiment should influence and what outcome you expect to see. Now you need to pick metrics
that measure if the experiment successfully achieves this outcome. An ideal success metric
directly measures the desired outcome and is:
- Observable in the short term
- Sensitive to changes
- Relevant for the business in the long term
In the best case, you can measure your desired outcome directly and with a reasonable delay after a
user's exposure to the change.
**Example**
Consider an example that makes a change in the user flow for subscribing to
premium. The experimenters can measure the share of users who successfully
sign up. The impact on user behavior is directly related to the change in the
user flow, it's measurable in the short term, and highly relevant to the
business.
Unfortunately, often the outcome of interest happens further in the future and is difficult
to measure directly in the experiment.
**Example**
For example, when we create a new feature at Spotify, we often hope to improve
the user experience and reduce churn in the long term. But the subjective user experience is difficult to
measure, and the impact of the user experience on churn takes time to detect. In those
cases, we need to use proxy metrics that we can measure in the short term, and are reliable predictors of
the long-term outcome that's our primary interest.
**Example**
At Spotify, common proxy metrics are *share of active users* (measured over a
day or a week) and *minutes played*. These metrics measure short-term
engagement with the product and correlate with long-term outcomes like churn
and premium subscription.
## Select few specific metrics
Success metrics should be as specific to the hypothesis as possible.
You may be curious to learn about all the possible effects that your treatment may have. It's often
tempting to just add every single metric that your change could possibly impact. However,
when deciding on a success metric you should limit yourself to a few relevant metrics,
and separate explorations from the criterion that defines success.
You should select only a few success metrics because:
- It's harder to reliably measure success with many metrics
- More metrics require a larger sample size
After your experiment ends, you can explore the effects on other metrics using exploratory analysis. This can
help you understand the results better and inspire new hypotheses. However, you should base the decision whether
to ship a change on your pre-defined success metrics, not on metrics that you
added afterwards. Pre-defining decision criteria helps to avoid
[confirmation bias](https://en.wikipedia.org/wiki/Confirmation_bias), where you end up selectively
looking for evidence that confirms your beliefs and ignore evidence against.
**In Confidence:** In Confidence, you can run [exploratory analysis](../../docs/exploration) after your experiment ends to dig deeper into results and get inspiration for new hypotheses.
### Example
**Example**
Consider a team that's working on the Spotify home page that wants to test whether adding a "shuffle" button in
the "Try something else" shelf increases user engagement on the home screen. They create an
experiment with two treatment groups: one called "Control" which gets the default experience (no
shuffle button), and one called "Treatment" which gets the shuffle button.
They need to decide on a success metric to decide whether the shuffle button improves user
experience. Which metric should they choose?
If the goal of the button is to increase interaction with the **Try something else** shelf, then one
possible metric is *Share of users who play from the **Try something else** shelf*. This
directly measures the behavior that the feature aims to influence. But is this also relevant for the
user and the business? Measuring success by that metric makes it tempting to introduce more features
that direct traffic towards this shelf, and away from the **Jump back in** and **Podcasts to try** shelves. A better
success metric is *Minutes played on Week 1*, because this measures overall user activity.
You could add *Share of users who play from the **Try something else** shelf* as a metric to confirm
that an increase in plays from **Try something else** caused an increase in overall activity.
## Use the minimum detectable effect to set the sensitivity of the experiment
After you decide which metric to use to measure success, you need to define what effect size you
want the experiment to be able to reliably detect. This effect size is called the "minimum
detectable effect" (MDE), or sometimes the "minimum relevant effect." You use the MDE to set up and
plan the experiment so that it has enough sensitivity to detect effects you consider meaningful.
Selecting the MDE is a trade-off between:
- the smallest business relevant effect
- the smallest practically measurable effect
As an experimenter, use your domain expertise and discuss with stakeholders what the smallest effect
that consider meaningful is. In the next step, you use the MDE to calculate what amount of traffic
you need to reliably detect this effect. If the sample size you need to measure the chosen MDE is
unrealistically large, then you need to adjust MDE upwards.
One way to understand the MDE of an experiment is to imagine your experiment as a microscope.
### Illustration: MDE is like the resolution of a microscope
Imagine looking at cells under a microscope. The minimum detectable effect of an experiment is
analogous to the resolution of a microscope. With a blurry, low resolution image you can see large
structures. If you are specifically interested in smaller structures, you need a higher resolution.
For even smaller structures you need an even higher resolution. In experiments, you can increase the
sensitivity by increasing the sample size. This allows detecting smaller changes.
### Watch this video to learn more about what the MDE is and what to consider when you set it
[Video: https://www.youtube.com/watch?v=lV50V6hj2ms]
**Quiz: What is the purpose of a success metric in an experiment?**
- To validate statistical assumptions
- Evaluate if the product change had the intended impact (correct)
- Check that the experiment is correctly configured
**Quiz: Why do we specify a Minimum Detectable Effect (MDE) for success metrics?**
- to ensure that the experiments works everywhere in the world
- To ensure that the metric has a valid definition for this experiment
- To clearly define success and to calculate the required sample size (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-course/target-audience
# Lesson 7: Target Audience
> You can set up your experiment to target a specific group of users, defined by
what's called the target audience of your test. The users you target are the
users you learn about and you need to consider the population of your
experiment when you interpret the results and try to generalize them.
When you set up an experiment, you often want to test your new feature or change on a specific group
of users—such as new users, users in certain markets, or users on certain platforms.
The group you want to include in your experiment is called targeting population or **Audience**.
## The users you target are the users you learn about
The hypothesis you formulated in the planning phase should describe what users you want to target
with your test. It's important to think about that *the only group of users you can draw conclusions
about is the group of users that you include in your experiment*.
For example, the results of an experiment that targets the iOS app are not directly transferable to
the Android app. Similarly, the results from an experiment run on users in Brazil don't directly
generalize to, for example, users in Germany. Likewise, the results from an experiment on users
visiting a particular page in your app is not directly transferable to *all* your users. This may
sound obvious, but sometimes the situation may be a little less obvious. You need to consider the
population of your experiment when you interpret the results.
## What you can target on
Inclusion criteria can flexibly define the target audience of the test based on attributes. You can
pass in any information you want when resolving a feature flag. Any information that you pass in can
be used as an inclusion criterion for your experiment. For example, if you pass in country or device
type, these can be used to create inclusion criteria.
**In Confidence:** In Confidence, the connection between your product and the experiment is via [feature flags](../../docs/flags/introduction). When you resolve a feature flag in your code, you pass in context attributes that Confidence can use for targeting. For any feature flag with usage, Confidence provides autocomplete with the attributes that have been available in those contexts in the Audience section of the experiment setup page. If you don't know what exists in your evaluation context, the person who added the feature flag to the code probably does.
**Quiz: What is a correct conclusions from an experiment targeting only Brazil?**
- It is hard to be sure about for which countries or regions the results are representative for regardless of the targeting
- These results are representative for the whole world because Brazil is similar enough to the rest of the world and whatever works there should work everywhere
- The results are representative for all countries with the same size as Brazil
- These results are representative for Brazil, conclusions about any other country or region cannot be drawn without further assumptions and analysis (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-course/why-you-should-experiment
# Lesson 1: Why you should experiment
> In this lesson, you learn when you should experiment. Run experiments to:
- Objectively test your own biased assumptions
- Avoid accidentally causing breakage while trying to improve
- Innovate fast by abandoning bad ideas early
- Establish a causal link between a product change and an outcome
## We experiment because we know that we have biases
As humans, we tend to look for evidence that supports what we already believe, a phenomenon known as
[confirmation bias](https://en.wikipedia.org/wiki/Confirmation_bias). To make matters worse, we also
have a tendency to overvalue the products that we built ourselves (also known as the
[IKEA effect](https://www.behavioraleconomics.com/resources/mini-encyclopedia-of-be/ikea-effect/)).
This means that if we want to know the true value of product changes for our users, we have to be
very careful to measure the impact in an unbiased and objective way, to avoid having our own beliefs
fool us.
## We experiment to avoid accidental breakage
Every time we change something about our product, we run the risk of accidentally causing negative
side effects. This could be an increase in latency or crash rates caused by a new feature. For
a mature product such as Spotify, it is much easier to unintentionally break the user experience than
to improve it. Without experimentation, small undetected decreases in performance can add up and
have a detrimental combined impact on the overall user experience. 
## We run experiments to innovate fast and abandon bad ideas early
The most important thing for most companies, Spotify included, is not to ship **a lot of changes**,
but to ship **the right changes**. To not release negative product changes is as important as to
release new positive changes to the product. Without testing our assumptions systematically on real
users in a real life setting, we risk investing a lot of development resources into product changes
that appeared promising at first, but didn't actually improve the user experience in a real life
setting.
## Experiments allow us to draw causal conclusions
Let's say that we are looking for ways to reduce churn for Spotify premium users. We could do an
analysis that compares users who churned with users who didn't. One result of such an analysis could
be that users who didn't churn experienced more app crashes than users who churned. Does this mean
that increasing the number of app crashes would reduce churn? Of course not. People who use the app
a lot are more likely to experience a crash, and are also less likely to churn.
Now let's imagine that we built a new feature, and we hope that it reduces churn for Spotify
premium users. In theory, we could just roll out the feature to everyone, check how many people are
using it, and then see if people who use the feature are less likely to churn. But would this tell
us if the feature actually reduces churn? No. Because just as with app crashes, a correlation
between more feature usage and less churn would not imply a causal link. To objectively measure the
value of our new feature, we need to find a way to isolate the impact of the feature from everything
else that can impact our metric of choice. The gold standard method for doing this is called a
"randomized controlled trial."
## Randomized controlled trials
Experiments split users into two (or more) groups by random assignment. The random
assignment makes sure that the groups are, on average, similar in all aspects except for the change
we want to test. If we randomly split all Spotify users into two groups, the two groups should be
very similar in terms of dimensions like demographics, connection speed, and music taste. One group gets the
status of a "treatment" group and receives the new feature. The other group receives the default
feature. We can then observe the users over time while they receive two different experiences, and
measure some outcome of interest, for example churn, daily activity, or the number of minutes
played. At the end of the experiment, we run a statistical test to calculate whether the
differences between the groups are larger than what we expect to see if there's no difference.
## The cost of experiments
Experiments aren't free. The main costs involved are that:
- It takes time to set up an experiment, wait for users to be exposed and analyze the results.
- If the change that you test is as beneficial as you hope, then the users in the control group miss
out on the improved experience until the end of the experiment.
- If a change makes the user experience worse, then some users receive a worse experience for as
long as the experiment runs.
## The cost of *not* running experiments
- You don't know if users respond to the product change in the way that you expect.
- You might have negatively impacted your users in unexpected ways. If you roll out many changes
without A/B testing them on real users, there might be negative impacts on system performance,
crash rates, and more that you fail to detect. Taken together, they can add up and seriously
impact the user experience.
- Without testing your assumptions on real users, you risk investing resources into
product changes that appear promising, but don't actually improve the experience in a real life
setting.
## Learn more
Watch this video to see
examples of different types of experiments for various common use cases.
[Video: https://www.youtube.com/watch?v=B6QFyLkWiP8]
**Quiz: Why should all changes that affect end-users be tested with A/B tests and/or rollouts?**
- To ensure that our changes have the effects we intended and detect unexpected side effects that might harm our end users and thereby our business. (correct)
- To inform other parts of the company what we are working on to ensure transparency.
- Experimentation is important in itself, because there could be no learning without it.
---
Source: https://confidence.spotify.com/bootcamp/intro-course/course-wrap-up
# Course wrap up
**Congratulations! You have finished this introductory course on experimentation!**
Thank you so much for taking this course and helping to build an even stronger experimentation
community at your company!
Go back to [my learning page](..) to keep learning!
---
## Interpreting experiment results
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/introduction
# Welcome to *Interpreting experiment results*
*Interpreting experiment results* is an asynchronous, self-paced course that teaches you how to read and understand the results page in Confidence. By the end of this course, you will be able to look at any experiment results page and know exactly what every number, label, and recommendation means, and what to do with that information.
The course is designed to be accessible regardless of your background or role. You do not need a statistics degree to follow along. Where the details matter, this course explains them in plain language and flags where you can dig deeper if you want to.
Some things may look different from other tools you have used. Where Confidence does things its own way, the approach is grounded in years of iteration and [original research](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics).
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material. Complete each lesson's questions to track your
progress.
## Before you begin
This course works best if you have run at least one experiment in Confidence, or have followed the
[A/B test quickstart](../../docs/abtests/quickstart).
Having a concrete experiment in mind as you go through the lessons will help the concepts click.
## Lessons
This course consists of the following lessons:
- Lesson 1: The anatomy of the results page: Get oriented on the three sections of the results page and understand the basic logic connecting them.
- Lesson 2: The Spotlight: Understand the overall recommendation (Ship, Continue, End, or Abort) and what drives each one.
- Lesson 3: Means and relative effects: Understand what the control variant and treatment variant means represent, and why effects are shown as relative percentages.
- Lesson 4: Confidence intervals and precision: Learn what confidence intervals are, how to read them, and why their width tells you how precisely the effect has been measured.
- Lesson 5: Significance for success metrics: Understand what 'significant' and 'not significant' mean for success metrics, and how the CI position determines the status.
- Lesson 6: Guardrail metrics and NIMs: Learn how guardrail metric status labels work, what a non-inferiority margin is, and why it gives stronger evidence of safety.
- Lesson 7: Health checks and the SRM: Learn how Confidence verifies that your experiment is trustworthy, and what to do when a health check fails.
- Lesson 8: Variance reduction: Understand why the means shown in results may differ slightly from raw averages, and how to interpret them correctly.
- Lesson 9: Sequential and non-sequential tests: Learn when you can trust the results you see, and what your choice of evaluation strategy means for your experiment.
- Lesson 10: Exploratory analysis: Learn how to use explorations to learn more from your experiment without drawing false conclusions.
- Lesson 11: The winner's curse: Learn about the winner's curse, why significant results from underpowered experiments tend to overestimate the true effect, and how to use confidence interval precision as a practical safeguard.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/confidence-intervals-and-precision
# Lesson 4: Confidence intervals and precision
> In this lesson, you learn what confidence intervals are and how to read them. You learn that CI
width tells you how precisely the effect has been measured, and how sample size and metric noise
affect that precision. You also learn how to read CIs from sequential and non-sequential tests.
The relative effect shown for a metric (say, +4.2%) is a single number, often called a **point estimate**. It is your best estimate of the true effect of the treatment variant on that metric. But a single number cannot tell you how trustworthy that estimate is. That is what the **confidence interval** is for.
A confidence interval (CI) gives you two things at once: **where the effect likely is** and **how precisely you have measured it**. These are not separate ideas; they are two ways of reading the same interval.
**In Confidence:** In Confidence, each metric result is shown as a horizontal bar. The dot in the middle is the point estimate (the +4.2% in our example). The bar extends to the left and right of that dot, and those endpoints are the **lower bound** and **upper bound** of the confidence interval.
## Interpretation: correct or practical
Take an interval like [−15%, +6%]. The technically correct interpretation of a 95% CI is this: if you ran this experiment 100 times and computed a 95% CI each time, approximately 95 of those 100 intervals would contain the true effect. Notice that this is a statement about the **procedure**, not about the specific interval in front of you.
The practical interpretation you should use: **With high confidence, the true effect lies between −15% and +6%.** Reading it that way is not technically exact, but it leads to rational product decisions, and that is what matters. Read more about this in the [Notes for nerds](#z-tests-t-tests-and-the-bayesian-connection) section.
## CI width as precision
The width of the CI tells you how precisely you have measured the effect. A **narrow CI** means a precise estimate: you know roughly what the effect is. A **wide CI** means high uncertainty: the effect could be anywhere in a large range.
A wide CI that crosses zero is not just "not significant." It means **you do not yet know whether there is an effect at all.** If you have not yet reached the required sample size, the right response is to collect more data. If you have already met it, the wide CI is itself the finding: the effect is likely smaller than what the experiment was powered to detect, and concluding that nothing meaningful is happening may be exactly right. How required sample size is determined is covered in the [Sample Size I](../sample-size-calculation-i/introduction) course.
A bit simplified: two factors drive CI width in the most basic setting. **Sample size**: more users means a narrower CI. This is the most controllable factor; running longer or on a larger audience directly buys precision. **Metric variance**: some metrics are inherently noisier than others. Higher variance means you need more data to achieve the same precision. In practice, CI width also depends on whether you are running sequential or non-sequential testing, the number of metrics, the significance level (alpha), and other factors covered in the [Sample Size](../sample-size-calculation-i/introduction) and [Hypothesis Testing](../hypothesis-testing/introduction) courses.
Use the interactive below to build intuition for how these two factors interact.
[Interactive: CIWidthSimulator]
Try the following:
- Set sample size to 100 and σ to 40. Notice how wide and uncertain the CI is.
- Increase sample size to 5,000 with the same σ. The CI narrows significantly.
- Set sample size back to 1,000 and move σ from 5 to 50. The CI widens as the metric becomes noisier.
The key insight: **collecting more data is the primary lever you have to narrow the CI**. [Variance reduction (Lesson 8)](./variance-reduction) is another lever; it effectively gives you a narrower CI for the same sample size by removing predictable noise from the data. Choosing a less noisy metric in the first place is a third: metric sensitivity and how to assess it are covered in the [Feasibility and sensitivity](../intro-to-metrics/feasibility-and-sensitivity) lesson.
## CIs in sequential and non-sequential tests
The CI you see on a results page always reflects the data collected up to that moment. How to interpret it depends on the evaluation strategy the experiment uses.
With a **non-sequential** (fixed-horizon) test, the CI is only statistically valid at the pre-specified end point. Intervals you see mid-experiment have not been corrected for repeated looks. They are informational but should not be acted on.
With a **sequential** test, the CI is valid at any point in time. That is precisely what sequential testing is designed for. The interval starts deliberately wide when data is sparse, because the test distributes its false-positive budget across all future looks, and narrows as data accumulates, just as any CI does with more data.
**Reading a sequential result is straightforward: the CI at the latest time point is your current result.** Interpret it exactly like any other confidence interval. The time-series view shows how the result developed over the course of the experiment, but the actionable number is always the most recent one.
**In Confidence:** In Confidence, the CI bar and summary shown in the results always reflect the latest time point. Expand a metric row to see the full time-series plot of how the interval has evolved. Move the pointer across the graph to inspect the CI at any earlier point in time.
**Quiz: A confidence interval for a metric shows [-2%, +9%]. What is the best interpretation of this result?**
- The treatment variant definitely improved the metric by somewhere between -2% and +9%
- The true effect is plausibly anywhere from -2% to +9%, so we cannot be confident the change is positive (correct)
- The treatment variant had no effect because the interval includes zero
- The experiment needs to be stopped immediately
**Quiz: What does a wide confidence interval tell you about a metric result?**
- The treatment variant had a large effect on the metric
- The effect has been measured with low precision and you need more data to know what the true effect is (correct)
- The metric is not important enough to measure
- The treatment variant effect is large and statistically reliable
**Quiz: Which of the following actions directly leads to a narrower confidence interval?**
- Choosing a lower significance threshold (alpha)
- Adding more metrics to the experiment
- Collecting more data by running the experiment longer or on a larger audience (correct)
- Stopping the experiment early once results look promising
## Notes for nerds
### Two-sided intervals, one-sided tests
Although confidence intervals in Confidence are displayed as symmetric two-sided intervals, the hypothesis tests underlying them are always one-sided. Each displayed CI is best understood as two back-to-back one-sided confidence bounds: the lower bound is the one-sided lower bound (testing whether the effect is negative), and the upper bound is the one-sided upper bound (testing whether the effect is positive). The actual significance decision for any metric is made in a single, pre-specified direction: the direction in which the metric is intended to move. One-sided tests require fewer observations than two-sided tests to achieve the same power for a given effect size, which is one reason Confidence can detect effects with fewer users than tools that use two-sided tests by default.
### Z-tests, t-tests, and the Bayesian connection
Confidence uses z-tests throughout, not t-tests. For the sample sizes it targets (typically well above 1,000 users per group), the z and $t_{2n-2}$ quantiles are within 0.1% of each other, so the normal approximation costs nothing in practice and tracking degrees of freedom adds complexity for no gain.
Despite this, there is a precise argument for why Bayesian language is appropriate when interpreting results. The thread runs as follows. With a non-informative prior $p(\mu, \sigma^2) \propto 1/\sigma^2$ on each group, the Bayesian credible interval for the difference in means is numerically identical to the frequentist **t-test** confidence interval (see the derivation below). That Bayesian interval has a formal probabilistic guarantee: "there is a 95% probability that the true effect lies in this interval" is literally correct for it. The frequentist t-test interval converges to the z-test interval as $n$ grows, and by the Bernstein-von Mises theorem the Bayesian posterior converges to the same normal distribution. At the sample sizes Confidence targets, the z-test CI and the Bayesian credible interval are equal to any practically relevant precision, so the Bayesian probabilistic interpretation carries over.
The technically correct frequentist statement is: "95% of confidence intervals constructed using this procedure, across repeated experiments, would contain the true effect." This describes the long-run behavior of the procedure, not the probability that this specific interval is correct. For reading an experiment results page, treat the CI as a range of plausible values for the effect and you will make sound decisions.
**If you are comfortable saying "95% probability the effect is in this range" for a Bayesian interval with a non-informative prior, you can use the same language for the frequentist CI without meaningfully misleading yourself or your colleagues.**
The interactive below shows the convergence. Green is the frequentist z-test CI ($z = 1.96$, fixed); indigo is the Bayesian credible interval using $t_{2n-2}$, which starts slightly wider at small $n$ and converges to the z-test CI as sample size grows.
[Interactive: FreqBayesCI]
The full derivation of why the Bayesian credible interval equals the frequentist t-test CI, and why both converge to the z-test CI:
#### Step 1: Non-informative prior
For a single normal group with unknown mean $\mu$ and variance $\sigma^2$, the standard objective (Jeffreys) prior is:
$$p(\mu, \sigma^2) \propto \frac{1}{\sigma^2}$$
For two independent groups, control $(\mu_c, \sigma_c^2)$ and treatment $(\mu_t, \sigma_t^2)$, the joint prior is the product:
$$p(\mu_c, \sigma_c^2, \mu_t, \sigma_t^2) \propto \frac{1}{\sigma_c^2 \sigma_t^2}$$
#### Step 2: Marginal posterior for each group mean
For group $j$ with $n$ observations $x_{j1}, \ldots, x_{jn}$, the normal likelihood is:
$$p(\mathbf{x}_j \mid \mu_j, \sigma_j^2) \propto (\sigma_j^2)^{-n/2} \exp\!\left(-\frac{(n-1)s_j^2 + n(\bar{x}_j - \mu_j)^2}{2\sigma_j^2}\right)$$
Multiplying by the prior $1/\sigma_j^2$ and integrating out the nuisance parameter $\sigma_j^2$ yields the marginal posterior for $\mu_j$:
$$\mu_j \mid \mathbf{x}_j \sim t_{n-1}\!\left(\bar{x}_j,\ \frac{s_j^2}{n}\right)$$
where $\bar{x}_j$ is the sample mean and $s_j^2$ is the unbiased sample variance. The result is a Student's $t$-distribution rather than a normal, because integrating out the unknown variance introduces extra tail weight that shrinks with more data.
#### Step 3: Posterior for the difference
Under the assumption of equal variances ($\sigma_c^2 = \sigma_t^2 = \sigma^2$), the posterior for the difference $\Delta = \mu_t - \mu_c$ is:
$$\Delta \mid \mathbf{x}_c, \mathbf{x}_t \sim t_{2n-2}\!\left(\bar{x}_t - \bar{x}_c,\ s_p^2 \cdot \frac{2}{n}\right)$$
where the pooled sample variance is $s_p^2 = \frac{(n-1)s_c^2 + (n-1)s_t^2}{2n-2}$. The 95% credible interval is:
$$\hat{\Delta} \pm t_{2n-2} \cdot s_p\sqrt{2/n}$$
In the frequentist framework, the two-sample $t$-statistic under the same assumption is $T = (\bar{x}_t - \bar{x}_c) / (s_p\sqrt{2/n}) \sim t_{2n-2}$, giving the 95% confidence interval:
$$\hat{\Delta} \pm t_{2n-2} \cdot s_p\sqrt{2/n}$$
The formulas are the same. The frequentist CI and the Bayesian credible interval with a non-informative prior have numerically identical bounds.
#### Step 4: Convergence to normal
As $n \to \infty$, the Bernstein-von Mises theorem guarantees that the posterior converges to a normal distribution centered at the maximum likelihood estimate, regardless of the prior. Since $t_{2n-2} \to \mathcal{N}(0,1)$, both intervals converge to $\hat{\Delta} \pm 1.96 \cdot s_p\sqrt{2/n}$.
#### Informative priors
Informative priors can offer genuine value. If you have reliable prior evidence from previous experiments or strong domain knowledge, encoding it as an informative prior will shift and shrink the posterior, producing a narrower interval when that prior is accurate. That is a real statistical benefit. In practice, though, the benefit rarely justifies the complexity. Specifying a prior well requires choosing a distribution family, setting its parameters, and verifying that the inference is not too sensitive to those choices. There is also a deeper problem: for experiment results to change decisions, they need to travel from the analyst to the product manager, designer, and leadership who act on them. The bottleneck is rarely statistical precision. It is whether people trust and understand the evidence. As Sebastian Andersson discusses in [How Experimental Evidence Travels Through Your Organization](https://confidence.spotify.com/blog/experimental-evidence), that chain of inference is fragile, and adding layers of Bayesian machinery tends to obscure rather than clarify.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/exploratory-analysis
# Lesson 10: Exploratory analysis
> In this lesson, you learn how to interpret segmented results in Confidence. You will know
what a dimension result means, how to read it, and why segment findings require follow-up
experiments rather than direct decisions.
Splitting experiment results by user subgroups is powerful, but it requires a specific way of reading the results. This lesson focuses on one question: after you have split results by a subgroup, how do you interpret what you see?
**In Confidence:** In Confidence, you split results by user subgroups by adding **dimensions** to an exploratory analysis. For the broader context on exploratory analysis (what explorations are, how to create them, and why the false positive risk matters), see the [intro to metrics course](../intro-to-metrics/segment-analysis).
## What a dimension result shows
When you add a dimension, Confidence shows you the metric result for each segment separately. For example, if you split by platform, you see one result for iOS users and one for Android users. Each segment result has its own point estimate and confidence interval.
**Note:** Confidence always uses the dimension value from right before a user was exposed to the treatment variant. This means the segment a user is in cannot have been affected by the treatment variant itself. Static attributes like country or device type are inherently safe. Dynamic attributes like subscription tier or engagement level are also safe, because Confidence uses the pre-exposure snapshot.
## How to read a segment result
A segment result is a metric result like any other. The point estimate is the relative effect within that segment, and the confidence interval tells you how precisely that effect has been measured.
Read each segment result the same way you would read any CI:
- Is the effect in the expected direction?
- Does the CI cross zero? If it does, the effect for this segment is not statistically significant.
- How wide is the CI? Segments typically have fewer users than the full experiment, so the CI will be wider. A wide CI in a segment means you have less precision, not that the effect is different.
## What it means when segments look different
If one segment shows a positive effect and another shows no effect (or a negative effect), this is called a **heterogeneous treatment variant effect**: the treatment variant appears to work differently for different types of users.
Before drawing conclusions from a pattern like this, consider two things.
First, you are running one test per segment. With many segments, some will look significant by chance. A result that stands out across two or three segments is more credible than one that barely clears the threshold in a single segment.
Second, sample sizes per segment are smaller than for the full experiment. The difference you see between segments might be noise rather than a genuine interaction. Overlapping confidence intervals between segments are a strong sign that the apparent difference is not reliable.
The right response to an interesting segment pattern is not to ship only to that segment based on the exploratory result. It is to run a new experiment with that segment as the target population and the segmented metric as the pre-registered success metric.
**Example**
An experiment shows no significant improvement overall. When split by platform, iOS shows +4.2% (95% CI: [−0.5%, +8.9%]) and Android shows −1.1% (95% CI: [−6.0%, +3.8%]). Both CIs cross zero. Both results are non-significant. The overlap between the two CIs is large.
The pattern is suggestive, but neither result is significant and the CIs heavily overlap. This is not strong evidence of a genuine platform difference. It is worth noting as a hypothesis to test in a follow-up experiment, but not a reason to ship selectively to iOS users.
## Signs of a credible segment result
A segment result is more credible when:
- It was pre-specified before the experiment ran (you predicted this segment would respond differently).
- The effect size is large and the CI does not cross zero.
- It replicates the direction of the overall result, just amplified (rather than going in the opposite direction).
- It makes intuitive sense given what you know about the product and the treatment variant.
A segment finding that is surprising, post-hoc, and narrowly significant should be treated as a hypothesis, not a conclusion.
**Quiz: You split experiment results by country and notice that one country shows a significant positive effect while all others do not. What is the most appropriate response?**
- Ship the treatment variant to that country only, since you have a significant result
- Treat it as a hypothesis and run a new experiment targeting that country with this metric pre-registered (correct)
- Declare the overall experiment a success because at least one segment was positive
- Ignore the result because segment results are never valid
**Quiz: A segment result shows a wide confidence interval that crosses zero. What does this tell you?**
- The treatment variant had no effect on this segment
- The segment result is not statistically significant and the effect is measured with low precision, likely because the segment has fewer users (correct)
- The segment should be excluded from the analysis
- The experiment has an SRM for this segment
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/guardrail-metrics-and-nims
# Lesson 6: Guardrail metrics and NIMs
> In this lesson, you learn what guardrail metrics are and how to read their
status labels. You see how adding a non-inferiority margin (NIM) changes what
the labels mean, and why a NIM gives you stronger evidence of safety than no
NIM.
Guardrail metrics are not the ones you are trying to improve. They are the ones you want to make sure you do not damage. Examples: session length, error rate, revenue per user. The question for a guardrail is not "did this go up?" but "did this go in the wrong direction?"
Because the question is different from a success metric, the status labels are different.
## Without a NIM
The CI for a guardrail metric sits on the same axis as for a success metric. The zero line is still the reference. But the question is now about the harmful direction, not the positive direction.
For a metric where increases are harmful (such as time in checkout, error rate, or support contacts):
- **Has deteriorated**: the CI is entirely on the wrong side of zero, in relation to the improvement-direction of the metric. There is statistical evidence the metric moved in the harmful direction. This is a serious signal.
- **Has not deteriorated**: the CI crosses zero. No statistical evidence of movement in the wrong direction. The guardrail is holding.
Use the interactive below with the "No NIM" checkbox checked to explore these labels.
[Interactive: CIGuardrailSimulator]
Try the following with "No NIM" checked:
- With the direction set to "Decreases" (harmful = increase), drag the point estimate from +15% to -15% and watch both states: "Has deteriorated" when the CI sits entirely above zero, and "Has not deteriorated" when the CI crosses zero.
- Move it to +3% and reduce the sample size to 200. The wide CI crosses zero: "Has not deteriorated."
- Move the point estimate above +5% with a small sample size. The CI may sit entirely above zero: "Has deteriorated."
## What a NIM is
A non-inferiority margin (NIM) defines how much deterioration is acceptable. Rather than asking "did this metric move at all?", a NIM lets you say "we accept up to X% increase: anything within that tolerance is acceptable."
Adding a NIM changes the question from "did this harm?" to "did this stay within bounds?"
**In Confidence:** In Confidence, the NIM appears as a solid vertical line on the results page,
with an arrow pointing toward the safe zone. Whether a metric is non-inferior
or possibly inferior is determined by the CI's position relative to the NIM
line. **Has deteriorated** still uses the zero line as its threshold.
## With a NIM
- **Non-inferior**: the CI is entirely within the NIM boundary. This is positive evidence of safety: even the worst-case bound is within the acceptable tolerance.
- **Possibly inferior**: the CI crosses the NIM boundary. Not enough evidence to confirm the metric stayed within the tolerance.
- **Has deteriorated**: the CI is entirely on the wrong side of zero, in relation to the improvement-direction of the metric. Statistical evidence that the metric moved in the harmful direction, same threshold as without a NIM.
If a metric is possibly inferior, you can still interpret the CI the usual way: with high confidence, the true effect is somewhere between the lower and upper bound. That means you can use the CI to assess the worst case. Look at the bound in the harmful direction—the upper bound for a metric where increases are harmful, or the lower bound for a metric where decreases are harmful. That bound tells you how bad the effect could plausibly be, even if you cannot yet rule out that the metric stayed within the NIM.
To explore the with-NIM states, clear "No NIM" in the interactive above and try the following:
- With the direction set to "Decreases" (harmful = increase), drag the point estimate from +15% to -15% and watch all three states appear in sequence: "Has deteriorated" when the CI is entirely above zero, "Possibly inferior" as the CI crosses the NIM boundary, and "Non-inferior" when the entire CI is within the acceptable range.
- Set the point estimate near 0%. The CI sits well below the NIM: "Non-inferior."
- Move the point estimate toward the NIM value. The CI starts to cross the NIM line: "Possibly inferior."
- Move the point estimate well above the NIM. The whole CI is above zero: "Has deteriorated."
## The key difference
**Note:** Without a NIM, "Has not deteriorated" tells you only that you could not detect
harm: absence of evidence, not evidence of absence. With a NIM, "Non-inferior"
is positive evidence of safety. The CI is entirely within the acceptable
range, so even the most pessimistic estimate is acceptable. The second
approach is more rigorous when safety matters.
**Quiz: A guardrail metric where increases are harmful has a NIM of 5%. The CI shows [+0.8%, +3.4%]. What is the status?**
- Has deteriorated, because the point estimate is positive
- Possibly inferior, because the CI does not include zero
- Non-inferior, because the entire CI is within the NIM of 5% (correct)
- Possibly inferior, because the point estimate is positive
**Quiz: A guardrail metric where increases are harmful has no NIM. The CI shows [+0.9%, +11.2%]. What is the status?**
- Has not deteriorated, because the lower bound is below 1%
- Has deteriorated, because the CI is entirely above zero (correct)
- Possibly inferior, because the CI is wide
- Non-inferior, because no NIM was specified
**Quiz: Why is 'Non-inferior' a stronger result than 'Has not deteriorated' for a guardrail metric?**
- Because Non-inferior means the point estimate is exactly zero
- Because non-inferior means that there is evidence of limited harm. Has not deteriorated is just lack of evidence of harm (correct)
- Because Non-inferior experiments always have larger sample sizes
- Because Has not deteriorated is only used when the experiment is still running
## Notes for nerds
The distinction between "Has not deteriorated" and "Non-inferior" maps onto a broader framework for thinking about what level of evidence an experiment actually needs to provide. Rather than treating every experiment as requiring the same strength of evidence, you can think of experimentation as a ladder of risk mitigation: each rung offers progressively stronger statistical guarantees, but also requires more data to reach. Using guardrail metrics without NIMs sits at a lower rung: you are ruling out obvious harm, but not positively bounding how bad things could be. Adding a NIM moves you up the ladder: you are now producing positive evidence of safety within an acceptable tolerance.
This framing has practical implications for how you design experiments when sample sizes are limited. The [Confidence blog post on experimenting with smaller samples](https://confidence.spotify.com/blog/smaller-sample-experiments) develops this idea in full.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/health-checks-and-srm
# Lesson 7: Health checks and sample ratio mismatch
> In this lesson, you learn what the health checks on the results page are testing, why they matter,
and what to do when one of them fails. The sample ratio mismatch check is especially important:
if it fails, you cannot trust any of the metric results.
The health checks section sits between the Spotlight and the Metrics section on the results page. Its job is to answer a question that comes before any metric interpretation: **did this experiment actually run correctly?**
Metric results are only meaningful if the experiment was set up correctly and ran as intended. The health checks verify this, and they flag problems before you invest time reading individual results.
## Incoming traffic
This check verifies that your experiment is actually receiving traffic. Specifically, it confirms that the flag rule the experiment controls is being evaluated by clients, that those evaluations are being applied, and that exposure is being calculated for all groups.
If this check fails, the experiment has not been collecting data as expected. There is nothing to interpret yet.
## Balanced traffic (the SRM check)
This is the most important health check, and the one that most frequently requires action.
When users are assigned to experiment groups, they should be distributed according to the allocation you configured. If you set up a 50/50 split, then roughly half the users should be in control and half in the treatment group. If you set up a 33/33/33 split across three groups, each group should have approximately a third of the users.
The **sample ratio mismatch (SRM) check** tests whether the observed distribution of users across groups matches the expected allocation. If there is a meaningful imbalance (more users in one group than there should be), this is a strong signal that something went wrong in the implementation or exposure logic.
**Recommendation:** If the SRM check fails, stop the experiment and investigate before drawing any conclusions from the results. A sample ratio mismatch means the groups are likely not comparable, which invalidates all metric results. Results from an experiment with an SRM cannot be trusted, even if the metric numbers look good.
Why does a traffic imbalance invalidate the results? The entire logic of a randomized experiment depends on the groups being statistically equivalent before the treatment variant is applied. If the exposure logic has a bug or was implemented incorrectly (for example, if code sitting between the experiment and the SDK affects users in one group more than another), the groups may differ systematically in ways unrelated to the treatment variant. Any observed difference in metrics could then be due to that pre-existing difference, not the treatment variant.
Common causes of SRM include bugs in the assignment logic, caching issues that cause some users to miss the exposure event, and SDK integration issues where custom code between the experiment and the SDK causes uneven exposure. The SRM check will not tell you which of these is the cause: it will only tell you that something is wrong. Investigating requires looking at the exposure data and the experiment setup in detail.
## No metric deterioration
This check verifies that none of the metrics you are tracking have moved significantly in the wrong direction. This is a continuous check: it runs throughout the experiment using sequential tests regardless of which evaluation strategy you have chosen (more on that in [Lesson 9](./sequential-vs-non-sequential)).
If this check fails, at least one metric has deteriorated. The specific metric showing deterioration is listed in the health check section. This triggers an **Abort** recommendation in the Spotlight.
**Note:** The metric deterioration health check and the deterioration status labels on individual guardrail metrics are related but not identical. The health check is a broad, always-on, early-warning scan that applies to all tracked metrics. The guardrail metric status label is the formal statistical result for that metric based on the test configuration you set up. If either signals deterioration, pay attention.
## What the Abort recommendation means
When you see **Abort** in the Spotlight, Confidence is telling you that something has gone wrong that makes continuing the experiment harmful or pointless. This is either because a health check failed (typically SRM) or because one or more metrics have deteriorated significantly.
Aborting an experiment is not a failure. It is the system working as intended. Catching a problem early, before a damaging feature reaches all users, is one of the core reasons to run experiments in the first place. At Spotify, 42% of experiments are aborted because harm was detected, not because the experiment was poorly designed, but because the experimentation system did exactly what it should. That is an enormous amount of value protected. The [Spotify experiments with learning framework](https://engineering.atspotify.com/2025/9/spotifys-experiments-with-learning-framework) post has more on how Spotify thinks about this.
**Quiz: The balanced traffic health check fails for your experiment. What should you do?**
- Ignore it and focus on the metric results, since the metrics look good
- Stop the experiment and investigate the cause before drawing any conclusions from the results (correct)
- Double the experiment duration to collect more data
- Remove the underrepresented group from the analysis
**Quiz: Why does a sample ratio mismatch invalidate experiment results?**
- Because it means not enough users were exposed to the experiment
- Because it means the groups may not be comparable, so observed metric differences could be caused by the imbalance rather than the treatment variant (correct)
- Because it indicates a bug in the metric calculation logic
- Because it means the experiment ran for too short a time
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/reading-means-and-relative-effects
# Lesson 3: Means and relative effects
> In this lesson, you learn what the control variant mean and treatment variant mean represent, why Confidence reports effects
as relative percentages, and how to read the numbers shown for each metric.
For each metric in your experiment, Confidence shows you two numbers: the **control variant mean** and the **treatment variant mean**. From those, it calculates a **relative change**. Understanding what these numbers actually represent and why effects are shown as percentages makes everything else on the results page easier to read.
## Control variant mean and treatment variant mean
An experiment splits users randomly into groups. The control group receives the current experience. Each treatment group receives a changed experience. After the experiment runs for a period of time, Confidence computes the average value of each metric for the users in each group.
- The **control variant mean** is the average metric value across all users assigned to the control variant.
- The **treatment variant mean** is the average metric value across all users assigned to a given treatment variant.
Both numbers reflect what actually happened for real users during the experiment. They are not predictions or model outputs; they are averages of observed data.
**Example**
A metric tracks the number of items added to a cart per user. After two weeks, the control variant average is 2.45 items and the treatment variant average is 2.61 items. These are the control variant and treatment variant means shown in Confidence.
## Why relative effects?
Rather than showing the raw difference between the treatment variant and control variant means, Confidence always shows the **relative change**: the difference expressed as a percentage of the control variant mean.
$$
\text{Relative change} = \frac{\text{Treatment variant mean} - \text{Control variant mean}}{\text{Control variant mean}}
$$
So in the example above: (2.61 − 2.45)/2.45 ≈ +6.5%.
There are two good reasons for this:
**Comparability across metrics.** Your experiment may include metrics measured in completely different units: seconds, click rates, counts, revenue. A 0.5-second improvement means something very different for a 5-second process than for a 5-minute one. Relative effects put everything on the same scale, making it much easier to compare results side by side.
**Intuitive interpretation.** A +5% improvement in conversion rate is immediately meaningful to most people, regardless of whether the baseline is 2% or 20%. The absolute difference is harder to assess without knowing the baseline.
**Note:** Because the effect is expressed relative to the control variant mean, you always need the control variant mean to interpret the absolute magnitude of the change. A +5% change on a metric with a control variant mean of 0.01 is a much smaller absolute movement than a +5% change on a metric with a control variant mean of 1,000.
## What to look for
When you read the metric results in Confidence, start with three things:
1. **The direction.** Is the relative change positive or negative? Does that match what you expected from the treatment variant?
2. **The magnitude.** Is the effect large enough to matter in practice? A +0.01% change on a revenue metric might be statistically detectable but commercially irrelevant.
3. **The confidence interval.** This tells you the range of plausible values for the true effect. You will learn how to read it in the next lesson.
## A note on adjusted means
When variance reduction is active for a metric (which it is by default in Confidence), the control variant and treatment variant means shown are slightly adjusted versions of the raw group averages. This adjustment is designed to produce a more precise estimate of the treatment variant effect. You will learn exactly how this works in [Lesson 8](./variance-reduction).
For now, the key point is that you should interpret the relative change shown in Confidence in the same way regardless of whether variance reduction is active.
**Quiz: The control variant mean for a metric is 100 and the treatment variant mean is 107. What is the relative effect shown in Confidence?**
- +7 units
- +7% (correct)
- +0.07%
- It depends on the metric type
**Quiz: Why does Confidence report effects as relative percentages rather than absolute differences?**
- Because absolute differences are always misleading in experiments
- To make effects comparable across metrics measured in different units and at different scales (correct)
- Because the raw means are not available in Confidence
- To make the numbers look larger and more impressive
## Notes for nerds
Even though Confidence displays effects on a relative (percentage) scale, all statistical testing is done on the **absolute scale**. The relative effect shown is simply the absolute estimated difference divided by the control variant mean, a transformation applied after the test is run.
This matters because it means Confidence does not need to account for the control mean being an estimate rather than a fixed number. The inference is based on the absolute difference, which has clean frequentist properties. If significance testing were done on the relative scale directly, the control mean would itself be stochastic across repeated experiments, complicating the statistical guarantees. By testing on the absolute scale and only transforming for display, Confidence avoids this issue entirely.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/sequential-vs-non-sequential
# Lesson 9: Sequential and non-sequential tests
> In Confidence, sometimes you can see results during the experiment and sometimes only after it ends.
This lesson explains why, and what it means for how you act on what you see.
In Confidence, sometimes you can see results during the experiment and sometimes only after it ends. This depends on the evaluation strategy chosen for the experiment. Understanding the difference is important because it determines when you can act on what you see.
## The peeking problem
Standard statistical tests are designed to be run once: you collect your data, run the test, and look at the result. If you look at the result repeatedly (checking every day, or every time data updates) and you are willing to act the moment you see significance, you are effectively running many tests. Each look is an opportunity to find a false positive by chance, and the more often you look, the more likely you are to get one.
With a 5% false positive rate per test and no correction, checking results 20 times gives you roughly a 64% chance of seeing at least one false positive result across all those looks, even if there is truly no effect.
## Sequential tests: built for continuous evaluation
**Sequential tests** solve the peeking problem by adjusting the statistical test to account for the fact that you will look at the results multiple times. They allocate the acceptable false positive rate across all the looks you will take, so that the overall false positive rate stays controlled. Sequential testing is the right choice when speed matters: when you want to be able to ship as soon as you have strong enough evidence, without waiting for a pre-specified end date.
There are two main types of sequential tests:
- **Group Sequential Tests**: used when you provide an expected sample size. These are more powerful than always-valid tests and are the preferred option when you have a reasonable estimate of how many users your experiment will collect.
- **Always-Valid Tests**: used when you do not provide an expected sample size. These can run indefinitely without inflating the false positive rate, but they have lower power than group sequential tests for a given sample size.
The trade-off for both types is that sequential tests require slightly more data than non-sequential tests to achieve the same power for a given effect size.
**In Confidence:** In Confidence, you use sequential tests by selecting **Continuous** as the evaluation frequency. Confidence automatically picks between Group Sequential Tests and Always-Valid Tests based on whether you provide an expected sample size.
## Non-sequential tests: the highest power option
A standard non-sequential test (also called a fixed horizon test) is designed to be run once, after data collection ends. With this approach, you should not act on the metric results you see while the experiment is still running. The numbers mid-experiment are informational but they do not have the statistical guarantees of the final result.
The benefit is that non-sequential tests achieve higher statistical power for the same sample size compared to sequential tests. If you know in advance you will run the experiment for a fixed period regardless of what you see, this is the most efficient choice.
**In Confidence:** In Confidence, non-sequential tests correspond to the **Upon Conclusion** evaluation frequency. You can see the current powered effect at any point during the experiment, which helps you judge when it makes sense to end it.
**Note:** The recommendation at Confidence is to use upon-conclusion evaluation unless speed of decision is a priority. This is because most teams benefit from running experiments for a full, pre-specified period to account for time effects and novelty effects, and upon-conclusion evaluation maximizes power for that use case.
## Deterioration checks always run sequentially
Regardless of which evaluation strategy you choose, Confidence always monitors your metrics for deterioration using sequential tests. This means you will be alerted if a metric starts to move in the wrong direction, even if you chose upon-conclusion evaluation and have not yet reached the end of your experiment.
This is an important design choice: the ability to detect harm early is always on. You do not sacrifice protection against regressions by choosing a non-sequential test for your main results.
## What this means in practice
When you look at experiment results, the key question to ask is: what evaluation strategy is this experiment using?
- **Sequential (continuous) evaluation**: the results you see are statistically valid to act on at any point. The test has accounted for repeated looks.
- **Non-sequential (upon conclusion) evaluation**: the final result after the experiment ends is valid to act on. Mid-experiment numbers should be used for awareness only, not decisions.
For a deeper look at how to choose between these strategies when setting up an experiment, including trade-offs around time effects, novelty effects, and power, see [Lesson 2: Choose evaluation strategy](../../bootcamp/advancing-experimentation/choose-evaluation-strategy) in the Advance your experimentation course.
**Quiz: You are running an experiment with upon-conclusion evaluation. Why shouldn't you look at the results before the experiment has ended?**
- Because the experiment might not have reached the target audience size yet
- Because success metrics are only calculated once the experiment ends
- Because upon-conclusion testing only provides valid statistical guarantees at the predetermined end point — peeking and acting on interim results inflates the false positive rate (correct)
- Because early results always show larger effects that shrink as more data comes in
**Quiz: What is always true about deterioration monitoring in Confidence, regardless of evaluation strategy?**
- Deterioration is only checked when you select continuous evaluation
- Deterioration checks always use sequential tests, so you are protected against regressions at all times (correct)
- Deterioration checks are only run when the experiment reaches its required sample size
- You must manually trigger deterioration checks for upon-conclusion experiments
## Notes for nerds
Confidence supports two types of sequential tests. **Group Sequential Tests (GST)** are used when you provide an expected sample size at setup. They are more powerful than the alternative for a given sample size and are the preferred option when you have a reasonable estimate of how many users the experiment will collect. **Always-Valid Tests** are used when no expected sample size is provided. They can run indefinitely without inflating the false positive rate, at the cost of somewhat lower power than a GST for the same amount of data.
The sample size calculator in Confidence accounts for this choice. When you switch from non-sequential to sequential evaluation, you can see the impact on required sample size directly in the calculator. Sequential tests require more data to achieve the same power for a given effect size, and the calculator shows you exactly how much.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/significance-and-status-labels
# Lesson 5: Significance for success metrics
> In this lesson, you learn what the status labels on success metric results
mean and how to read a confidence interval to determine whether a result is
significant. You also learn what "not significant" really means, and why it is
not the same as "no effect."
Every metric result in Confidence has a **status label**: a short phrase that tells you what to conclude about that metric.
## Success metrics: did the treatment variant improve things?
For success metrics, Confidence is asking: _is there statistical evidence that the treatment variant changed this metric in the desired direction?_
The zero line on the results page is the reference point. A confidence interval that sits entirely on the positive side of zero means the data are inconsistent with there being no effect: the result is statistically significant. A CI that crosses zero means "no effect" remains a plausible value.
- **Significant**: the CI does not cross zero in the direction of the test. There is statistical evidence that the treatment variant moved the metric in the intended direction. This is not certainty; it means the data are unlikely to look the way they do if there were truly no effect.
- **Not significant**: the CI crosses zero. There is not enough statistical evidence to conclude that the treatment variant affected this metric. The data are consistent with there being no effect.
**Note:** "Not significant" does not mean "no effect." It means "no evidence of an
effect strong enough to detect with the current data." With a wide confidence
interval, you simply do not yet have enough data to know. Do not interpret
"not significant" as proof that the treatment variant did nothing.
Use the interactive below to build intuition for how the CI position determines significance.
[Interactive: CISuccessSimulator]
Try the following:
- With the direction set to "Increases", drag the point estimate from +15% to -15% and watch all three states appear in sequence: "Has improved" when the CI clears zero on the positive side, "Not significant" as it crosses zero, and "Has deteriorated" when the CI sits entirely below zero.
- Move the point estimate back to a moderate positive value, then set the direction to "Decreases." The same sweep now works in reverse: "Has improved" appears on the negative side, and "Has deteriorated" on the positive side.
- Set sample size to 500 and move the PE to +5%. The wider CI may still cross zero even with a positive estimate.
- Increase sample size to 5,000. The CI narrows, and a smaller PE becomes sufficient for significance.
## A note on adjusted significance thresholds
When an experiment has multiple success metrics, the significance threshold is corrected to control the overall false positive rate across all metrics. Adding more success metrics makes each individual metric slightly harder to call significant. This is the statistically correct approach: without it, the more metrics you add, the more likely you are to find a false positive by chance.
**In Confidence:** In Confidence, when an experiment has multiple success metrics, the
significance threshold is automatically adjusted for each metric to control
the overall false positive rate. In the Detailed results view, each metric
shows its own **adjusted alpha** value.
**Quiz: A success metric (where increases are desirable) shows a +3.1% change with a status of 'Not significant'. What is the correct interpretation?**
- The treatment variant definitely had no effect on this metric
- There is not enough statistical evidence to conclude the treatment variant changed this metric (correct)
- The experiment should be stopped because the effect is too small
- The effect is too small to be worth measuring
**Quiz: A success metric CI shows [+1.2%, +6.8%]. The metric improves with increases. What is the status of this result?**
- Not significant, because the CI is narrow
- Significant, because the entire CI is above zero (correct)
- Not significant, because the point estimate is below 5%
- Cannot be determined without knowing the sample size
## Notes for nerds
### Two decisions, two false positive rates
What looks like a single significance test is actually two distinct decisions, each answering a different question. The **shipping decision** asks: is there enough evidence that this treatment variant improves the metrics we care about? The **abort decision** asks: is there evidence that this treatment variant is causing harm right now, and should we stop the experiment early to prevent it?
These are not the same question, and there is no statistical reason to answer them using the same false positive rate. The consequences of getting them wrong are asymmetric. A false positive on the shipping decision means you ship a variant that does not actually improve things, which is a recoverable mistake. A false negative on the abort decision means you keep running an experiment that is harming users, which is a much more urgent problem. It follows that the abort decision should use a more sensitive threshold: you want to catch deterioration early, even if it means occasionally stopping an experiment that would have recovered.
There is also a fundamental difference in how time enters each decision. The shipping decision is made at a single point in time (when the experiment ends), which is exactly the setting upon-conclusion evaluation is designed for. The abort decision, by contrast, must be made continuously throughout the experiment: you cannot wait until the end to find out whether users were harmed. This means the abort decision should always use a sequential test, regardless of which evaluation strategy was chosen for the shipping decision. Upon-conclusion evaluation gives you no valid way to act on mid-experiment results; a sequential test is specifically designed for exactly that.
Confidence implements this separation by default. The deterioration check that feeds into the Abort recommendation always runs as a sequential test, even when the experiment uses upon-conclusion evaluation for its main results. The shipping decision and the abort decision are evaluated independently, with different statistical properties suited to each. The theoretical grounding for this approach is developed in [Schultzberg, Ankargren, and Frånberg (2024)](https://arxiv.org/abs/2402.11609).
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/the-anatomy-of-the-results-page
# Lesson 1: The anatomy of the results page
> In this lesson, you learn how the results page is organized and the basic logic that connects its three sections.
You also learn why a single experiment can produce many individual results and how Confidence helps you navigate them.
When you open the results page for an experiment in Confidence, the first thing to notice is that it has a clear structure. There are three sections, stacked from top to bottom: **Spotlight**, **Health checks**, and **Metrics**. Each section serves a different purpose. Understanding what each one answers is the key to reading results efficiently.
## The three sections
### Spotlight
The Spotlight is at the very top of the page. It gives you an **overall recommendation** for each treatment variant: whether to ship, continue, end, or stop the experiment. Think of the Spotlight as a summary that synthesizes everything below it into a single actionable signal.
You do not need to read every metric result to know whether your experiment is worth shipping. The Spotlight does that job. Later in this course you will learn exactly what drives each possible recommendation.
### Health checks
The health checks section serves two related purposes. First, it verifies whether the **experiment itself is trustworthy**: that traffic is flowing as expected and that groups are balanced. Second, it monitors whether **any metric is moving in the wrong direction**, which can trigger an Abort recommendation even when the experiment is technically set up correctly.
In other words, health checks are not only about data quality. They are also an active monitoring layer that can tell you the treatment variant is causing harm before you have finished collecting data.
### Metrics
The Metrics section shows the **results for each individual metric**, for each treatment variant compared to the control variant. This is where you see the actual numbers: the estimated effect, confidence intervals, and whether the result is statistically significant.
## Many results from one experiment
A single experiment can produce a large number of individual results. If you have `T` treatment variants (not counting the control variant) and `M` metrics, you have `T × M` individual results to interpret. For example, an experiment with two treatment variants and four metrics gives you eight results.
**Example**
An experiment tests two new versions of a checkout flow (version A and version B) against the current version (control variant), using four metrics: conversion rate, time in checkout, number of errors, and customer satisfaction score.
That gives 2 treatment variants × 4 metrics = **8 individual results** to interpret.
This is one reason the Spotlight is so useful: it collapses those eight results into two recommendations, one per treatment variant.
## How the sections connect
The three sections work together. Health checks feed into the Spotlight: if a health check fails, the Spotlight takes that into account. The Metrics section provides the raw signal that the Spotlight summarizes. As you work through the course, you will learn each section in detail. By the end, you will be able to trace any Spotlight recommendation back to the individual numbers that produced it.
**Quiz: An experiment has 3 treatment variants and 5 metrics. How many individual results are there to interpret in the Metrics section?**
- 5
- 8
- 15 (correct)
- 3
**Quiz: What is the primary purpose of the Spotlight section?**
- To show the raw metric values for each group
- To check whether the experiment received enough traffic
- To give an overall recommendation that summarizes all metric results (correct)
- To display the experiment setup and design
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/the-spotlight
# Lesson 2: The Spotlight
> In this lesson, you learn what each Spotlight recommendation means, what drives each one,
and how the Spotlight synthesizes many metric results into a single actionable signal.
The Spotlight is the first thing you see on the results page. It gives a recommendation for each treatment variant: **Ship**, **Continue**, **End**, or **Abort**. Each recommendation has a precise meaning, and understanding what drives each one lets you immediately understand the state of your experiment when you open it.
## The four recommendations
### Ship
**Ship** means Confidence has found sufficient evidence that the treatment variant is worth rolling out. Three conditions must all be true for a Ship recommendation:
1. At least one success metric has improved significantly in the intended direction.
2. All guardrail metrics meet their tolerance levels: either significantly non-inferior (if you use non-inferiority margins), or showing no evidence of deterioration (if you do not).
3. No health check has failed. There is no SRM and no evidence of metric deterioration.
Ship is a positive signal, but it is a recommendation, not a mandate. You should still exercise judgment. Consider whether the effect size is practically meaningful, whether the experiment ran long enough to rule out novelty effects, and whether the results make sense given your understanding of the product.
### Continue
**Continue** means there is not yet enough evidence to ship, and no reason to stop. This is the most common recommendation for an experiment that is on track. It means:
- No success metric has significantly improved yet.
- No health checks have failed.
- No metrics have deteriorated.
The right response to **Continue** is to let the experiment run until it reaches the required sample size. Stopping an experiment early because results are not yet significant is a common mistake: it produces biased, unreliable results.
### End
**End** appears when an experiment has collected enough data to be considered **powered** for all success metrics, but none of those metrics has shown a significant improvement. This is different from **Continue**.
**Continue** means "we do not know yet." **End** means "we have collected enough data, and there is no signal."
An **End** recommendation is a genuine null result. The treatment variant does not appear to improve the metrics you care about, and you have enough data to be fairly confident in that conclusion. The right response is to stop the experiment and treat this as a real finding: the treatment variant did not work as hypothesized.
**Note:** A null result is a valuable result. Knowing that a change did not improve metrics saves engineering and design resources that would otherwise go into shipping and maintaining a change that does not help users. Do not dismiss null results.
### Abort
**Abort** means something has gone wrong that makes continuing the experiment harmful or pointless. This recommendation appears when:
- A health check has failed (most commonly a sample ratio mismatch), which means the results cannot be trusted.
- One or more metrics have deteriorated significantly, meaning there is statistical evidence the treatment variant is harming something you care about.
When you see **Abort**, stop the experiment. If the cause is an SRM, investigate the exposure logic before relaunching. If the cause is metric deterioration, the treatment variant may be harmful and should not be shipped.
## After the experiment ends
After you end an experiment, the **Continue** and **End** recommendations merge into a single **Don't ship** label. The **Abort** recommendation may also appear if there was a health check failure. The **Ship** recommendation remains if the evidence for shipping was already established before the experiment ended.
## The full picture
The Spotlight is a synthesis. It takes the outcomes of all success metrics, all guardrail metrics, and all health checks, and compresses them into one recommendation per treatment variant. Understanding the individual components (significance, CI width, health checks, evaluation strategy) gives you the ability to look at a Spotlight recommendation and trace it back to its causes.
**Example**
An experiment shows a **Continue** recommendation in the Spotlight, with one success metric showing "Not significant" (+1.4%) and all guardrail metrics showing "Has not deteriorated." The experiment is halfway through its planned duration.
The correct interpretation: the experiment is healthy and on track. The success metric has not yet moved enough to be statistically significant, but "not significant" at the halfway point is expected. Continue running until the required sample size is reached before drawing conclusions.
**Quiz: The Spotlight shows 'End' for a running experiment. What is the most accurate interpretation?**
- The experiment has been aborted due to a health check failure
- The experiment has collected enough data and no success metric has improved significantly. This is a genuine null result (correct)
- The experiment ran out of traffic and cannot continue
- The treatment variant improved all metrics significantly
**Quiz: Which of the following must be true for Confidence to recommend 'Ship'?**
- All success metrics must be statistically significant
- At least one success metric has improved significantly, all guardrail metrics are acceptable, and no health check has failed (correct)
- The experiment has reached twice its required sample size
- All metrics (success and guardrail) must show no deterioration
## Notes for nerds
The Spotlight's synthesis of many metric results into a single recommendation is a non-trivial statistical problem. When you have many metrics, each tested at some significance level, the probability that at least one shows a spurious significant result grows quickly. Spotify has published research on how to make principled, risk-aware product decisions under exactly this kind of multi-metric setting. The underlying ideas are described in a paper by Schultzberg, Ankargren, and Frånberg (2024). You can read the engineering post at [engineering.atspotify.com](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics).
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/type-m-and-type-s-errors
# Lesson 11: The winner's curse
> In this lesson, you learn about the winner's curse: the systematic tendency of statistically
significant results from underpowered experiments to overestimate the true effect. You learn what
Type M and Type S errors are, why neither can be diagnosed after the fact, and how actual power
diverges from planned power when the true effect differs from the MDE.
You already know about two ways an experiment can go wrong: a **false positive** (Type I error) and a **false negative** (Type II error). There are two more failure modes that only appear *inside* statistically significant results. They are subtler, and they tend to go unnoticed precisely because the result looks like a success.
This lesson builds on statistical power and significance. If you need a refresher on those concepts, see the [Hypothesis testing](../hypothesis-testing/introduction) course and the [Sample size calculation](../sample-size-calculation-i/introduction) courses before continuing. For the formal statistical definitions, see [Gelman & Carlin (2014)](https://journals.sagepub.com/doi/10.1177/1745691614551642).
## The winner's curse
Imagine running many experiments on a small true effect with insufficient sample size. Most experiments will *not* reach statistical significance. The signal is too weak relative to the noise. The ones that *do* reach significance got there by chance: they happened to draw an unusually large estimated effect. Only extreme estimates cross the significance threshold.
This is the **winner's curse**. Your significance filter does not just select for real effects; it selects for *large-looking* estimates. When an experiment is underpowered, the results you actually see and act on are not a representative sample of all possible results. They are the lucky outliers.
## Two new types of errors
If you took the [hypothesis testing course](../hypothesis-testing/introduction), you might remember the two classic error types: the [Type I error](../hypothesis-testing/fpr-and-alpha) (a false positive, controlled by α) and the [Type II error](../hypothesis-testing/tpr-and-power) (a false negative, controlled by power). The winner's curse produces two more, named by statisticians Andrew Gelman and John Carlin, that only appear inside statistically significant results:
**Type M error (Magnitude)** is the tendency for significant results to overestimate the true effect size. The *exaggeration ratio* tells you by how much: it is defined as average significant effect/true effect. On average, a significant result from an underpowered experiment reports an effect much larger than what is actually there.
**Type S error (Sign)** is the risk that a significant result points in the entirely wrong direction. It is rare except in severely underpowered experiments. In practice, this means either aborting an experiment even though the treatment had a positive effect, or shipping a treatment even though it had a harmful effect.
Both errors are conditional on significance. They describe what happens inside the pool of experiments that find significant results, not the full distribution of experiments.
## Explore both risks
The simulator below shows the sampling distribution of effect estimates, for a given true effect and sample size. The **green area** represents the fraction of all experiments that would find a significant result in the correct direction (your statistical power). The **red area** is the fraction that would find a significant result pointing the *wrong* way (Type S error). The stat cards update as you move the sliders.
The simulator assumes a metric that improves when it increases, and a true effect in the direction of improvement. The Type S risk shown is therefore the risk of finding a significant deterioration and aborting an experiment that is actually positive.
[Interactive: TypeMSExplorer]
Notice the key patterns:
- **Lower true effect** (move the left slider down): Type M overestimation grows sharply. At small true effects, a significant result might report an effect that is 5× or 10× the truth.
- **Smaller sample size** (move the right slider toward Small): the distribution widens, the green area shrinks (lower power), the red area grows (higher Type S risk), and the overestimation gets worse.
- **Type S error stays small** unless the experiment is extremely underpowered. It only becomes a meaningful concern at very low power with tiny true effects.
## Planned power versus actual power
The explorer above treats the true effect as your only input. In practice, you size an experiment around a *minimum detectable effect* (MDE), the smallest effect worth caring about, and that choice determines your planned power. If the true effect is smaller than the MDE, actual power is lower than planned and Type M inflation is worse than you expected. If the true effect is larger than the MDE, actual power exceeds your target and the overestimation shrinks.
The simulator below separates the two. Set the MDE and the true effect independently to see how actual power diverges from planned, and how the Type M factor changes as the gap grows.
[Interactive: TypeMSPlot]
When the true effect equals the MDE, actual and planned power match. Slide the true effect below the MDE and actual power drops, and the significant results you do get come from the lucky tail of the distribution and overestimate the truth by more.
## The fundamental problem: the true effect is never known
Both simulators above ask you to specify the true effect. In real life, you never know this. You run the experiment precisely *because* you do not know whether, or by how much, the treatment works.
This single fact changes how to use the concepts above. Type M and Type S are **design-time sensitivity tools**, not post-hoc diagnostics:
- Before you run an experiment, you can ask: *if the true effect were around X%, what would the exaggeration ratio be?* That question is useful for planning sample size and setting expectations.
- After you see a significant result, you cannot work backwards to say whether your specific estimate is inflated. A reported effect of +3% could be a Type M-inflated estimate of a true +0.6% effect, or a slight underestimate of a true +4% effect. You cannot tell from the data alone.
- Even at the conventional 80% power, if the true effect exactly equals your MDE, the expected overestimation is around 13%.
The picture is not all bad. Type S errors are rare outside of extremely underpowered settings, so directional decisions from experiments are generally reliable: if your experiment says the treatment helps, it very likely does. The more practical concern is Type M, which affects effect size estimates even in reasonably powered experiments. If you need an accurate magnitude, the best remedies are running a larger experiment, or following up a significant A/B test with a gradual rollout, which both replicates the finding and yields a larger-sample estimate of the true effect size under real traffic conditions.
**Quiz: An underpowered experiment reaches statistical significance and reports a +4% effect. What does the winner's curse suggest about this result?**
- The result is a false positive and should be disregarded
- The true effect is likely larger than +4%
- The +4% estimate is probably an overestimate of the true effect (correct)
- The experiment should be re-run with more data before any conclusion is drawn
**Quiz: You get a statistically significant result. Can you calculate the Type M exaggeration ratio for this specific result?**
- Yes — divide the observed effect by the minimum detectable effect
- Yes — it equals one divided by the statistical power
- No — Type M requires knowing the true effect size, which is always unknown (correct)
- No — Type M only applies when the p-value is below 0.01
**Quiz: An experiment is designed with 80% power to detect the MDE. The true effect turns out to be half the MDE. What happens to the Type M exaggeration ratio compared to what was planned for?**
- It stays the same, because the sample size did not change
- It decreases, because a smaller true effect is easier to detect accurately
- It increases, because actual power is lower than planned, so significant results come from a more extreme tail (correct)
- It cannot be determined without knowing the observed p-value
## Notes for nerds
### The winner's curse is a risk, not a certainty
A significant result does not only come from underpowered experiments detecting small effects. It can just as well come from a well-powered experiment detecting a large effect, and in that case the estimate is reliable. The winner's curse applies conditional on being underpowered relative to the true effect. If the true effect is large and the experiment was well-powered to detect it, the significant result is not inflated.
### The limits of "just be powered"
A widely quoted piece of advice is: don't trust results from underpowered experiments. This is sound, but it conceals an important subtlety.
Power is always defined *relative to an assumed effect size*. An experiment is "80% powered" to detect a 2% lift, or "30% powered" to detect a 1% lift. The same experiment can be well-powered for a large effect and poorly powered for a small one.
The question is not only *is this experiment powered?* but *powered for what effect, and how plausible is that effect?* An experiment designed to detect a 5% effect when the realistic effect is 0.5% is not trustworthy even if it reaches significance: it would require massive overestimation to do so. Conversely, a nominally underpowered experiment that finds a large effect might be reporting something accurate if the true effect really is large.
### Type S flips with the direction of the true effect
The simulator fixes the true effect as positive (an improvement). When that is the case, a Type S error means detecting a significant deterioration on a treatment that actually helps, and aborting prematurely.
If the true effect were negative (a harmful treatment), the roles reverse. Now a Type S error means detecting a significant improvement and shipping something that is actually harmful. The same rare-but-severe warning applies, just in the other direction.
### Empirical shrinkage
Some tech companies with strong experimentation programs use historical data to work around the unknown-true-effect problem. If you have run thousands of experiments and know the distribution of typical effect sizes for your product area, you can use that historical distribution as a prior and shrink new estimates toward it, pulling inflated estimates back toward more plausible values. It is statistically elegant, but it requires a large archive of past experiments with consistent metrics, and confidence that the historical distribution still describes the current product and user base.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/variance-reduction
# Lesson 8: Variance reduction
> In this lesson, you learn what variance reduction is and why it causes the means shown in Confidence
to differ slightly from raw group averages. The treatment variant effect is interpreted in the same way
as without variance reduction. The adjustment simply makes the estimate more precise.
When you look at the results for a metric in Confidence, the control variant mean and treatment variant mean shown may not be exactly the same as the raw averages for those groups. This is because **variance reduction** is active by default for most metrics. Understanding what this adjustment does and what it does not do is important for reading results correctly.
## The core idea: use pre-experiment data to reduce noise
Every metric has natural variation. Some users will use a feature a lot; others will barely touch it. Much of this variation has nothing to do with the experiment. It reflects pre-existing differences between users that existed before the experiment started.
Variance reduction works by using each user's **pre-experiment behavior** on the metric to predict and cancel out this pre-existing variation. Specifically, Confidence looks at how each user behaved on the metric *before* they entered the experiment, and uses that data to produce a more precise estimate of the treatment variant effect.
Think of it this way: if you know that certain users were already heavy users before the experiment, you can account for that when estimating whether the treatment variant changed their behavior. Without this adjustment, that pre-existing variation adds noise to your estimate. With it, much of that noise is removed.
**Note:** The pre-experiment data used for variance reduction must come from before the user entered the experiment, so it cannot be influenced by the treatment variant. This is what makes the adjustment valid.
## What changes and what does not
Because of the adjustment, the control variant and treatment variant means shown in Confidence are not the raw group averages. They are slightly shifted versions that have been adjusted to account for pre-existing differences between groups.
However, **the estimated treatment variant effect and the way you interpret it remain the same.** The point estimate (the relative % change) and the confidence interval are still your best estimate of the observed treatment variant effect. The adjustment makes them more precise, not different in meaning.
The relative change is calculated as the adjusted treatment mean minus the adjusted control mean, divided by the unadjusted control variant mean, so the percentage you see is still relative to the actual (unadjusted) control baseline.
**Example**
A metric shows a control variant mean of 195.8 and a treatment variant mean of 196.5 in Confidence. These are adjusted values. The raw group averages might have been 196.1 and 196.7. The relative change (+0.36%) and the confidence interval are computed from the adjusted values and represent a more precise estimate of the observed treatment variant effect than you would get from the raw averages alone.
## The variance reduction percentage
The variance reduction for a metric tells you how much of the original variance was removed by the adjustment. A variance reduction of 60% means that the adjusted estimate has 60% less variance than the raw estimate, effectively similar to having 2.5 times as many users without the adjustment.
A high variance reduction means the pre-experiment data was strongly predictive of post-experiment behavior. A variance reduction of 0% means no adjustment was applied.
**In Confidence:** In Confidence, you can see the variance reduction percentage for each metric in the Detailed results view.
Use the interactive below to see how variance reduction narrows the confidence interval compared to no adjustment, for the same sample size and metric noise.
[Interactive: CIWithVRSimulator]
Try the following:
- Set variance reduction to 0%. This is the CI you would get without any adjustment.
- Increase variance reduction to 60%. Notice how the CI narrows: that is extra precision from pre-experiment data, with no additional users.
- Now try increasing the sample size instead. Both approaches narrow the CI; variance reduction is the free version.
## Why this matters
Variance reduction is one of the main reasons sample sizes in Confidence can be smaller than in tools that do not use this technique. It improves the precision of every estimate without requiring more users. When you see a narrow confidence interval for a metric, variance reduction is often a contributing factor.
You do not need to think about variance reduction when reading results. Just know that the adjustment is there, it makes estimates more reliable, and you interpret the numbers the same way you would without it.
**Quiz: When variance reduction is active, the means shown for a metric in Confidence are...**
- The raw group averages, unchanged
- Adjusted values that account for pre-experiment differences between users, producing a more precise estimate (correct)
- Predictions from a machine learning model
- Averages computed only from users who were highly active before the experiment
**Quiz: How should you interpret the relative % change for a metric when variance reduction is active?**
- You should apply a correction factor before interpreting it
- You cannot interpret it directly. You need to look at the unadjusted raw means instead
- In the same way as without variance reduction. It is the estimated treatment variant effect, just measured more precisely (correct)
- As a lower bound on the observed treatment variant effect, since variance reduction tends to underestimate effects
## Notes for nerds
### The regression adjustment
The variance reduction method used in Confidence is covered in detail in the
[variance reduction lesson in the intro to metrics course](../intro-to-metrics/variance-reduction),
and its effect on required sample sizes is covered in the
[sample size calculation III course](../sample-size-calculation-iii/variance-reduction).
In short, the method fits separate regressions of the post-experiment outcome on the pre-experiment variable for each group, then adjusts the treatment variant effect estimate accordingly. The classical CUPED formulation uses a single adjustment coefficient for both groups. Fitting separate regressions per group is never worse, and is strictly better whenever users respond differently to treatment—which is the typical case. The two are equivalent only in a perfectly balanced 50/50 experiment [(Negi and Wooldridge, 2020)](https://www.tandfonline.com/doi/full/10.1080/07474938.2020.1824732).
### Bounds on covariate selection
In principle, any pre-experiment covariate can be included in the regression to reduce variance further. In practice, the pre-experiment measurement of the metric itself is hard to beat—and the gains from going beyond it are bounded. Even the most sophisticated feature engineering can narrow the confidence interval by at most a further 29% beyond what the simple pre-experiment metric already achieves [(Ting and Hung, 2023)](https://arxiv.org/abs/2311.17858).
### Adjusted control means in multi-variant experiments
One consequence is that in experiments with multiple treatment variants, the variance-adjusted control variant mean shown for a given comparison uses only the data from the variants involved in that specific comparison. This means the adjusted control variant mean can differ slightly between comparisons. This is expected and correct. It does not indicate an error in the data.
---
Source: https://confidence.spotify.com/bootcamp/interpreting-experiment-results/course-wrap-up
# Course wrap up
**Congratulations! You have finished *Interpreting experiment results*!**
You can now open any experiment results page in Confidence and know exactly what you are looking at. To recap what you have covered:
- The results page has three sections: **Spotlight**, **Health checks**, and **Metrics**, each answering a different question.
- The **control variant and treatment variant means** are averages of what actually happened for real users, and effects are always shown as **relative % changes** to make them comparable across metrics.
- A **confidence interval** tells you both where the effect likely is and how precisely you have measured it. A wide CI means you need more data, not that there is no effect.
- **Status labels** differ between success and guardrail metrics because they answer different questions: "did it improve?" versus "did it break anything?"
- The **SRM check** is the most critical health check. If it fails, no metric result can be trusted.
- **Variance reduction** makes estimates more precise by using pre-experiment behavior to remove noise. The numbers look slightly adjusted, but you interpret them the same way.
- Your choice of **evaluation strategy** determines when results are valid to act on. Deterioration checks always run sequentially regardless of that choice.
- The **Spotlight** synthesizes everything (health, success metrics, and guardrail metrics) into one recommendation per treatment variant.
- **Explorations** are for learning and hypothesis generation, not for deciding whether an experiment succeeded.
## What to explore next
If you want to go deeper on the statistical foundations behind what you learned here, the
[A primer on hypothesis testing](../hypothesis-testing/introduction)
course covers the mechanics of how hypothesis tests work and where p-values and significance thresholds come from.
To learn about more advanced experiment configurations, including guardrail metrics with non-inferiority margins and how to choose between sequential and non-sequential tests, check out
[Advance your experimentation](../advancing-experimentation/introduction).
Go back to [my learning page](..) to keep learning!
---
## Intro to metrics
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/introduction
# Welcome to *Intro to metrics*
*Intro to metrics* is an asynchronous, self-paced course that teaches you the fundamental concepts of metric design and selection. This course focuses on building intuition for what makes a good metric and how to design metrics that drive better product decisions.
In this course, you learn how to define metrics that capture the right user behavior, choose appropriate measurement approaches, and understand the different types of metrics used in experimentation and product development. Whether you're evaluating experiments, tracking product performance, or setting team goals, this course provides the foundation for working effectively with metrics.
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material. Complete each lesson's questions to track your
progress.
## Before you begin
Before you start this course, you should go through the
[A/B test quickstart](../../docs/abtests/quickstart) to familiarize yourself with the basics of running experiments in Confidence. While this course is platform-agnostic, understanding the experiment workflow helps you see how metrics fit into the broader context of product development and experimentation.
## Lessons
This course consists of the following lessons:
- Lesson 1: What is a metric?: Define metrics and understand their role in product decisions and experiments.
- Lesson 2: Metric roles: Understand success, guardrail, exploratory, and diagnostic metric roles.
- Lesson 3: Time considerations: Choose appropriate time windows for short-term and long-term impacts.
- Lesson 4: Capturing behavior: Design metrics that capture behavior without gaming or unintended effects.
- Lesson 5: Strategic metrics: Understand KPIs, proxy metrics, and the strategic metric hierarchy.
- Lesson 6: Interpretability: Create understandable metrics with clear naming and documentation.
- Lesson 7: Feasibility and sensitivity: Evaluate feasibility, variance, and influenceability for experiments.
- Lesson 8: Variance reduction: Understand regression adjustment, how much variance reduction to expect, and when to cap.
- Lesson 9: Select metrics: Apply your learnings from previous lessons and practice selecting a complete metric suite.
- Lesson 10: Segment-level analysis: Break down experiment results by user segments to find meaningful patterns across groups.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/capturing-behavior
# Lesson 4: Capturing behavior
> In this lesson, you learn how to design metrics that accurately capture the user behavior you care about. You explore common pitfalls to avoid—including filters that hide failures—understand the difference between direct and proxy metrics, learn about the danger of optimizing directly for proxies, and discover how to ensure your metrics truly reflect your hypothesis.
## Aligning metrics with user behavior
A good metric directly measures the user behavior you want to understand or
influence. This sounds simple, but it's surprisingly easy to choose metrics that
capture something different from what you intended.
The key question to ask when defining a metric is: **"If this metric moves, do I
know what user behavior changed?"**
Consider a team trying to improve product recommendations on an e-commerce site.
They might instinctively reach for "impressions of recommended products" as
their metric. It's easy to measure and seems relevant. But impressions only tell
you how many people saw the recommendations, not whether they found them useful.
A user could scroll past recommendations dozens of times without ever clicking.
The metric moves, but you don't know if recommendations improved.
A better choice would measure the actual discovery behavior: did users add
recommended products to their cart or wish list? This captures not just exposure
or even clicks, but genuine interest—the behavior that matters.
## Four common pitfalls
Four patterns repeatedly trip up teams when choosing metrics: confusing
activity with value, picking metrics that can be gamed, overlooking
unintended consequences, and filtering in ways that hide failures.
### Activity that doesn't indicate success
The first pitfall is measuring activity that doesn't indicate success. Take
search quality as an example. If you use "number of searches" as your success
metric, more searches might seem like success. But the opposite could be
true—more searches often means users can't find what they want on the first try.
An improved search experience might actually decrease searches because users
find what they need faster.
**Example**
**The search quality trap:**
You're improving search quality and choose "number of searches" as your success
metric. The experiment launches, and searches increase by 15%. Success?
Not necessarily. Analysis reveals users are performing multiple searches to find
what they want, whereas before they found it on the first try. The metric
improved, but the experience degraded.
**Better metric**: "Share of searches resulting in a play"—this captures whether searches actually led to the desired outcome.
### Metrics that can be improved without better experience
The second pitfall is choosing metrics that can be improved without improving
the experience. If your metric is "number of feature impressions," you could
increase it by showing the feature more frequently or more prominently, even if
users find it intrusive. The metric improves, but the experience degrades.
Measure "share of users who engaged with the feature when shown" instead—this
captures actual interest, not just exposure.
### Unintended consequences
The third pitfall is overlooking unintended consequences. Optimizing purely for
"time on site" might lead you to make navigation more difficult or force users
through extra steps, keeping them on the site longer while degrading their
experience. Time on site matters, but only when it reflects genuine engagement.
Pair it with quality indicators like "task completion rate" or "return visit
rate" to ensure you're measuring the right thing.
### Filters that hide failures
The fourth pitfall is filtering in ways that erase evidence of harm. If you're
testing a backend optimization and filter your latency metric to successful
requests only, a 4× increase in timeouts simply disappears from your
analysis—the metric improves while the experience degrades. The fix is pairing
a filtered success metric (latency among successful requests) with an unfiltered
guardrail (overall request success rate).
**In Confidence:** Confidence lets you configure metrics to pad missing values with zero rather than excluding them, which preserves evidence of failures rather than silently dropping them.
A related trap is post-treatment filtering: when the treatment itself changes
who enters the filtered population. If a new notification algorithm sends fewer,
higher-confidence notifications, click-through rate among recipients may
improve—but you're measuring a cherry-picked group, not a real improvement in
engagement. When the treatment determines who gets filtered in, the metric is
compromised by design.
## Direct metrics versus proxy metrics
Sometimes you can measure exactly what you care about. Subscription conversions
measure the actual business outcome. Purchase completion measures the
transaction you care about. Customer satisfaction surveys measure actual
sentiment. These are **direct metrics**: they capture the outcome itself, not a
substitute.
But direct metrics aren't always practical. You might care about long-term
retention, but you might only have a few weeks to run an experiment. Or you care about customer
satisfaction, but don't have survey infrastructure in place. This is where
**proxy metrics** come in. These measure something related to your outcome,
serving as a stand-in.
**Example**
**When to use a proxy:**
You want to measure whether a new SaaS product feature creates more long-term
active users. But your experiment needs to conclude in 2 weeks to meet a release
deadline.
**Direct metric** (what you really care about): Monthly active users over 6 months
**Proxy metric** (what you can measure): Weekly active users and feature adoption in first 2 weeks
The proxy works if you've validated that users who adopt the feature in the
first two weeks tend to remain active long-term. Document this correlation so
others understand what the metric represents.
### The danger of proxy optimization
Proxy metrics are useful for measuring outcomes quickly, but they become dangerous when you start optimizing directly for them instead of the real outcome they represent. This is the risk of **gamification**—when you can improve the proxy without improving what you actually care about, the relationship breaks down.
Consider a streaming platform like Spotify. The real goal is long-term retention—keeping users coming back month after month. But retention is slow to measure, so you look for proxies. You might discover that users who like songs have much higher retention rates. This makes sense: liking a song signals intent to return and listen again. Number of liked songs becomes a useful proxy for retention.
The proxy works well for observational analysis and for experiments that indirectly affect liking behavior. But the moment you start directly optimizing for liked songs—making the like button larger, prompting users to like more often, adding like suggestions—you break the relationship. You can easily increase likes without increasing retention because you've changed what liking means. Users who casually click a prominent like button aren't signaling the same intent as users who seek out the like feature.
The proxy has been gamed. It no longer correlates with retention because you've optimized for the proxy itself rather than the underlying behavior it represented.
**Example**
**A proxy compromised:**
**Initial observation**: Users who like songs have 40% higher 90-day retention than users who don't.
**Conclusion**: Number of liked songs is a good proxy for long-term retention.
**What works**: Testing new discovery features and measuring whether they increase likes (along with other engagement metrics). The relationship holds because users are liking songs for the same reasons they always did.
**What breaks**: Making the like button 3x larger and adding "Did you like this?" prompts after every song. Likes increase 60%, but 90-day retention doesn't budge. The proxy no longer predicts retention because you've changed user behavior—users now like songs casually rather than intentionally. The correlation is broken.
**Recommendation:** Proxy metrics are often genuinely necessary—no team can run a six-month experiment to observe long-term retention directly. The value of a proxy is that it makes outcomes measurable on an experimentally practical timescale. Use proxies for measuring outcomes, not for direct optimization. When you want to improve the real outcome (like retention), test changes that should affect the outcome directly, and use the proxy to measure results quickly. The critical discipline is validating with historical data that the proxy actually correlates with the outcome you care about—and revisiting that validation as your product and user base evolve.
## Hypothesis alignment
Your hypothesis should drive your metric choice, not the other way around. Start
with what you expect to happen and why, then choose metrics that would confirm
or refute your expectations.
Imagine you're adding a one-click checkout option to your e-commerce site. Your
hypothesis might be: "This will increase purchase completion rate because it
reduces friction in the checkout process." This hypothesis points you toward
specific metrics: purchases using one-click checkout (the direct behavior),
overall purchase completion rate (the intended outcome), and some quality check
like return rate to ensure the faster checkout doesn't lead to more impulse
purchases that customers regret.
**Note:** If you find yourself changing your hypothesis to match an available metric, stop and reconsider. It's better to invest in measuring the right thing than to optimize for a convenient but misleading metric.
## Key questions to ask
When defining a metric, ask yourself these questions:
1. Does this metric directly measure the behavior I care about, or is it a proxy?
2. If I'm using a proxy, could I improve it without improving the real outcome I care about?
3. If this metric improves, am I confident the user experience also improved?
4. Does this metric reflect what my hypothesis says should change?
5. Can someone artificially inflate this metric without improving the real outcome?
6. Could optimizing for this metric have negative side effects?
If the answer to question 2, 5, or 6 is "yes," you're at risk of measuring the
wrong thing. Proxies are useful for measurement but dangerous for direct
optimization.
**Quiz: Which metric best captures actual user interest in a feature?**
- Number of times the feature was displayed
- Number of clicks on the feature
- Share of users who used the feature multiple times (correct)
**Quiz: What is the main risk of using number of searches as a success metric for search quality?**
- More searches might indicate users cannot find what they want (correct)
- It is too difficult to measure
- Search data is unreliable
**Quiz: When should you use a proxy metric instead of a direct metric?**
- When the direct metric is unavailable or takes too long to measure (correct)
- When you want to make the metric easier to understand
- Proxy metrics should always be preferred over direct metrics
**Quiz: What should you do if an available metric does not match your hypothesis?**
- Run the experiment without a proper success metric
- Change your hypothesis to match the available metric
- Invest in measuring what truly reflects your hypothesis (correct)
**Quiz: Why is it dangerous to optimize directly for a proxy metric like number of liked songs?**
- Proxy metrics are less accurate than direct metrics
- You can artificially inflate the proxy without improving the real outcome it represents, breaking the correlation (correct)
- Proxy metrics take longer to measure than direct metrics
## Notes for nerds
**Proxy metrics in practice.** Spotify has done a lot of research and thinking about how to build proxy metrics that remain valid over time, how to validate them empirically, and how to detect when the proxy-outcome relationship has drifted. [This blog post](https://confidence.spotify.com/blog/proxy-metrics) covers practical approaches to proxy metric design and validation.
**Post-treatment filtering as a causal inference problem.** The lesson frames post-treatment filtering as a metric design pitfall and recommends pairing a filtered metric with an unfiltered guardrail. That fix is practical, but it addresses the symptom rather than the cause. When you filter on a variable that is itself affected by the treatment—for example, filtering to "users who received a notification" when the treatment changes notification volume—you are conditioning on a descendant of the treatment. In causal inference terms, this introduces selection bias: the filtered population is no longer comparable between treatment and control, because the treatment changed who entered the filter. The unfiltered guardrail helps you detect that something went wrong; diagnosing what requires understanding why the filter correlates with treatment assignment.
**Goodhart's Law and Campbell's Law.** The phenomenon described in this lesson—where optimizing for a proxy corrupts its relationship with the underlying outcome—has a name: Goodhart's Law. Originally observed by economist Charles Goodhart (1975) in the context of monetary policy targets, and later generalized by Marilyn Strathern: *"When a measure becomes a target, it ceases to be a good measure."*
The related **Campbell's Law** (Donald Campbell, 1976) extends this to social contexts: *"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."*
In machine learning and reinforcement learning, the same failure mode is called **reward hacking** or **specification gaming**—an agent finds a way to maximize the reward signal without achieving the intended goal.
These aren't just theoretical concerns. Most "growth hack" failures and metric manipulation scandals in product companies are applied examples of Goodhart's and Campbell's Laws. Designing metrics that are robust to gaming—and separating the metrics you measure from the metrics you optimize—is one of the most important practical skills in experimentation.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/feasibility-and-sensitivity
# Lesson 7: Feasibility and sensitivity
> Learn how to evaluate metrics for experiments by assessing feasibility, understanding that effective variance (after variance reduction) matters more than raw variance, recognizing that binary metrics aren't inherently more sensitive than continuous metrics, and ensuring your product change can actually influence the metric you're measuring.
## Feasibility: Can we measure it?
**Feasibility** means you have the data, infrastructure, and resources to
compute the metric reliably. Before committing to a metric, verify: **data
availability** (events are logged at the right granularity with sufficient
history), **technical capability** (you can compute it efficiently within
required time frames), and **sample size requirements** (you can collect enough
data at acceptable cost).
For example, "streams per user from playlist recommendations" is feasible with
logged events and user-level data. In contrast, "user satisfaction with
recommendation quality" requires expensive survey data that may not exist.
## Sensitivity: Can we detect changes?
A metric is **sensitive** if it reliably detects meaningful changes. Sensitivity
depends on two independent factors: **variance** (how much noise exists) and
**influenceability** (whether your change can actually move the metric). Both
must be favorable.
## Variance: Raw versus effective
Higher variance requires more users or longer runtime to detect a given effect
size. However, what matters is not **raw variance** but **effective variance**
after applying variance reduction techniques. The next lesson covers variance reduction in detail—for now, understand that the right covariate can dramatically change which metric is actually the most sensitive choice.
This distinction is critical. Variance reduction techniques use regression
adjustment to control for pre-experiment user behavior. These techniques (like
CUPED, CUPAC, and others) leverage temporal correlation: how stable the metric is
over time. A metric with high raw variance but strong temporal correlation can
end up with lower effective variance than a metric with moderate raw variance
but weak correlation.
**Example**
Consider two metrics for measuring engagement:
**Metric A - Total streams per user**:
- Raw variance: Very high (some users stream 1 song, others stream 1,000+)
- Temporal correlation: Very high (heavy streamers stay heavy streamers)
- Variance reduction factor: 80%
- Effective variance: Low
**Metric B - Did user create a playlist?**:
- Raw variance: Moderate (binomial metric)
- Temporal correlation: Low (playlist creation is sporadic)
- Variance reduction factor: 20%
- Effective variance: Moderate to high
After applying variance reduction, Metric A might actually have lower effective variance and detect changes faster, despite its higher raw variance.
### Binary versus continuous metrics
Converting continuous metrics to binary (changing "total streams" to "did user
stream more than 10 times?") has critical trade-offs:
Binary metrics can have lower variance when users cluster far from the
threshold, but they **lose information about magnitude**. Say you convert "streams per user" into a binary metric: "did the user stream more than 10 times?" Then a change from 15 to 50 streams produces the same binary outcome (1→1) as no change at all—both users were already above the threshold. You only detect crossings of that threshold, making binary metrics **less responsive to changes tested in the experiment**. Substantial behavior improvements may not register. Additionally, variance is highest when proportions are near 50%, so poor thresholds can increase variance.
**Note:** Binary metrics aren't inherently more sensitive. They can reduce variance but also reduce influenceability by ignoring non-threshold changes. Consider whether information loss justifies potential variance reduction—and whether regression adjustment techniques would make the continuous metric more sensitive anyway.
The techniques used to reduce effective variance—regression adjustment (CUPED, CUPAC), capping, and related methods—are covered in depth in [Lesson 8: Variance reduction](./variance-reduction).
## Influenceability: Can your change move the metric?
Even with perfect variance properties, a metric is useless if your product
change cannot move it. **Influenceability** measures whether your experiment can
actually affect the metric—this is completely independent of variance.
Influenceability depends on three factors: **Exposure** (what proportion of
users encounter the change?), **Mechanism** (does the metric measure behavior
the change can affect?), and **Effect size** (is the change [substantial enough
to impact the metric](https://confidence.spotify.com/blog/two-questions)?).
**Example**
Testing a playlist creation flow redesign:
**Low influenceability**: Using "monthly active users" as your primary metric.
Most users never create playlists, so the change can't influence them. Even a
perfect redesign might show no effect.
**High influenceability**: Using "playlist creation rate among users who start
the creation flow." All measured users are exposed, the metric directly captures
what you're improving, and UI changes can meaningfully affect completion.
Scoping your experiment population and choosing metrics aligned with your
intervention are crucial—perfect variance properties mean nothing if your change
can't move the metric.
## Trade-offs in practice
You've now seen that good metrics require feasibility, low effective variance,
high influenceability, interpretability, and business alignment. No single
metric optimizes all dimensions—you must make deliberate trade-offs:
**Sensitivity versus business alignment**: Metrics directly tied to business
value (revenue, long-term retention) often have high variance or long
measurement windows. Use sensitive proxies (engagement, short-term behavior) as
primary metrics while monitoring business metrics secondarily. Ask: do
improvements in my sensitive metric translate to business outcomes?
**Granularity versus variance**: Granular metrics (total revenue per user)
capture effect magnitude but have high variance. Before simplifying through
capping or binary conversion, check whether regression adjustment can give you
both granularity and sensitivity.
**In Confidence:** Confidence automatically applies variance reduction using regression adjustment. You can see how much variance was reduced for each metric on the detailed results page.
**Scope versus influenceability**: Broad metrics (platform retention) align with
business goals but resist movement from single experiments. Narrow metrics
(feature engagement) are movable by experiments but miss broader impacts. Solution: use
narrow, movable primary metrics for decisions while monitoring broad
metrics for unintended effects.
**Example**
Testing a new recommendation algorithm:
**Success metric**: "Streams from recommendations per user"—directly influenced
by your change, moderate variance with variance reduction, measures what you're
improving.
**Guardrail metrics**: Total streams (broader impact), premium conversion
(business outcome), recommendations shown (diagnostic).
This lets you decide confidently while understanding broader implications.
**Quiz: Why might a metric with high raw variance be better for experiments than a metric with low raw variance?**
- If the high-variance metric has strong temporal correlation, regression adjustment techniques can reduce its effective variance below the low-variance metric (correct)
- High variance metrics are always more sensitive
- Raw variance does not matter for experiment design
**Quiz: What is a key disadvantage of converting a continuous metric to binary?**
- Binary metrics are harder to compute
- Binary metrics always have higher variance
- Binary metrics lose information about magnitude and only detect changes when users cross the threshold (correct)
**Quiz: What makes a metric highly influenceable for a specific experiment?**
- Low variance and strong temporal correlation
- High proportion of users exposed to the change, direct causal mechanism, and sufficient effect size (correct)
- Simple calculation and easy interpretation
## Notes for nerds
**Ratio metrics and variance reduction.** Ratio metrics require special care not just for variance estimation, but also for variance reduction. When applying regression adjustment to a ratio metric, you can't treat the ratio as a simple scalar—the numerator and denominator each carry independent variance, and the covariance between them matters. This is covered in depth in [Lesson 8: Variance reduction](./variance-reduction).
**Heterogeneous treatment effects.** Influenceability describes the average effect of your change across the measured population—but that average can mask enormous variation. A feature might be highly influenceable for power users who interact frequently with the affected surface, while being essentially unmovable for casual users who rarely encounter it. Conversely, a change can produce a neutral average because it helps some segments while harming others, with the two effects cancelling out. Whether this matters depends on your product strategy. If your goal is aggregate improvement, the average effect is what counts. If you want to understand who benefits and why, average influenceability is not enough—segment-level analysis is the right tool, and it is covered in [Lesson 10: Segment-level analysis](./segment-analysis).
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/interpretability
# Lesson 6: Interpretability
> In this lesson, you learn how to create metrics that stakeholders can understand and act on. You explore clear naming conventions, documentation best practices, and techniques for communicating metric meaning to both technical and non-technical audiences.
## Why interpretability matters
A metric is only useful if people can understand what it means and what actions
to take when it moves. The most sophisticated metric in the world won't drive
good decisions if stakeholders can't interpret it.
When a metric changes, interpretability means everyone understands what user
behavior changed, whether the change is good or bad, what might have caused it,
and what actions to consider next. Without this shared understanding, you can't
make confident decisions.
Compare "adjusted engagement index" to "average session duration per active user
per week." The first leaves people guessing—what's being adjusted? What counts
as engagement? How do I interpret a 5% increase? The second is clear: you're
measuring how long active users spend in sessions each week. If it goes up,
users are spending more time. If it goes down, they're spending less.
## Metric name format
Metric names should be descriptive and self-explanatory. A good metric name
answers three questions: What are you measuring, for whom, and over what time
period?
The "what" is the behavior or outcome: purchases, clicks, conversions, completed
tasks. The "who" is the unit of analysis: per user, per session, per account. The
"when" is the time window: daily, in the first week, during the trial period. Put
these together and you get names like "purchase completion rate per user in
first 30 days" or "average pages viewed per session for returning visitors."
**Example**
**Building interpretable names:**
**Vague**: "Engagement metric v2"
What's being measured? What does v2 mean? Impossible to interpret.
**Clear**: "Average time spent per active user per week"
Immediately clear: you're measuring how much time active users spend weekly.
**Vague**: "W1RR for MAU"
Abbreviations create barriers, especially for new team members.
**Clear**: "Week 1 retention rate for monthly active users"
Anyone can understand this without a glossary.
### Write helpful descriptions
Every metric should have complete technical documentation—calculation logic, time windows, filters, and more. But your colleagues shouldn't need to look it up every time they see your metric. A well-written description saves everyone time.
Think of the description as your chance to help colleagues quickly understand
what the metric measures and why it matters. When someone is choosing metrics
for an experiment or reviewing results, they can read your description and
immediately know if this metric is relevant, without diving into the full
definition.
**In Confidence:** In Confidence, every metric has a dedicated definition page with all the technical details. The description field is what colleagues see without clicking through, so it's the first thing they use to assess relevance.
**Example**
**Metric name**: Purchase completion rate per user in first 30 days
**Weak description**: "Measures purchases"
This forces colleagues to click through to understand what's actually being
measured.
**Strong description**: "The percentage of new users who complete at least one
purchase within 30 days of signing up. Used to measure onboarding effectiveness
and early conversion."
This gives colleagues enough context to decide if the metric is relevant without needing to dig deeper.
A good description answers three questions: What behavior are you measuring? Why
does it matter? When would someone use this metric? The time you invest in
writing a clear description pays off many times over as colleagues reuse your
metric.
This matters even more as AI agents become part of experiment workflows. An agent can read the full metric definition, but when there are thousands of metrics to search through, a well-written description is far more useful than a definition: it lets the agent find the right metric by understanding its purpose rather than parsing its implementation.
Good descriptions are also the foundation of a **centralized metric documentation system**. When every metric has a clear definition, consistent calculation logic, and well-written context, the whole organization can share a single source of truth. Teams can discover existing metrics rather than recreating them, and when business needs change, the logic only needs to be updated in one place.
**In Confidence:** In Confidence, every metric you define is available to your entire organization—so the quality of your documentation directly affects how confidently your colleagues can reuse your work.
## Audience communication
Different stakeholders need different levels of detail. Engineers need technical
specifics: data sources, joins, edge cases, implementation notes. Product
managers need to understand what the metric measures, why it matters for the
product, and how to interpret changes. Executives need business impact,
connections to strategic goals, and what actions to consider.
The same metric can be explained three different ways. For an engineer: "We
calculate monthly active users as distinct user IDs with at least one logged
event in the trailing 30-day window, excluding test accounts and automated
traffic." For a product manager: "Monthly active users tells us how many unique
people used the product in the last month. It's our primary measure of active
user base size." For an executive: "Monthly active users grew 3% this quarter,
driven by strong growth in emerging markets and improved retention in the free
tier."
## Technical accuracy and comprehension
Sometimes the most technically accurate description is too complex for broad communication. When this happens, use simple language for general communication, provide technical details in documentation, and highlight any important caveats.
**Example**
**What engineers see internally**: The calculation involves log-transformation, variance reduction using CUPED, and handles edge cases for sessions under 10 seconds.
**What the metric is named**: "Average session length"
**The principle**: Metric names describe the behavior being measured. Statistical implementation details—transformations, variance reduction methods, edge case handling—belong in the technical documentation, not in the name that stakeholders use to understand results.
**Quiz: Which metric name best follows the interpretability principles taught in this lesson?**
- User engagement score
- Average sessions per active user per week (correct)
- Weekly active users (WNA)
**Quiz: What is the primary purpose of writing a strong metric description in Confidence?**
- To save colleagues time by helping them quickly assess if the metric is relevant (correct)
- To meet documentation requirements
- To provide all the technical calculation details
**Quiz: Why should metric names avoid including statistical implementation details like CUPED adjustments?**
- Statistical methods should be documented in code comments instead
- CUPED is automatically applied to all metrics in Confidence
- Metric names should focus on what behavior is being measured, not how the statistics are computed (correct)
**Quiz: A metric called 'support ticket volume' increases by 20%. Why is this hard to interpret?**
- Support tickets are difficult to track accurately
- The increase could mean either more customer problems or more customers using the product (correct)
- Volume metrics are inherently unreliable
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/metric-roles
# Lesson 2: Metric roles
> In this lesson, you learn about the different roles metrics play in experiments: success metrics that define what winning looks like, guardrail metrics that protect against unintended harm, diagnostic metrics that validate your experiment ran correctly, and exploratory metrics that help you understand why results occurred. You learn how to construct a balanced metric suite that supports confident decision-making.
## Why metric roles matter
Not all metrics serve the same purpose in an experiment. Some tell you whether to ship your change. Others help you understand what happened and why. Still others act as safety checks, ensuring you haven't broken something critical along the way.
The key to running effective experiments is understanding these different roles and selecting the right mix of metrics before you start. A metric that's perfect for understanding user behavior might be terrible for making a shipping decision. A metric that's essential as a guardrail might distract you if treated as a success measure.
When you define clear roles for each metric upfront, you create a framework for making decisions based on evidence rather than intuition or confirmation bias.
## Success metrics
**Success metrics** define what winning looks like. These are the metrics you use to decide whether to ship your change. Everything else in your experiment supports this core question: did we succeed?
A good success metric directly measures the behavior your hypothesis predicts will improve. If you're testing a faster checkout flow because you believe it will increase purchase completion, your success metric should measure purchase completion. If you believe a new feature will increase user engagement, your success metric should capture that specific type of engagement.
Success metrics need to be sensitive enough to detect changes that matter but stable enough to trust. They should be clearly interpretable—when the metric moves, you should know whether that's good or bad without needing complex analysis. And they should be few in number, typically one to three. More than that and you're not being clear about what success actually means.
**Example**
**An e-commerce company tests a simplified checkout flow:**
**Hypothesis**: Reducing checkout from five steps to three will increase purchase completion by making the process less frustrating and faster to complete.
**Success metric**: Purchase completion rate per session in the first 7 days
This metric directly measures the intended outcome. When it moves, you know whether your simplification worked. It's specific enough to be meaningful but broad enough to capture the full effect on purchasing behavior.
**Recommendation:** Select all your metrics during experiment planning, before you collect any data. Pre-defining your success criteria and guardrails helps you avoid confirmation bias and ensures you're measuring what you set out to learn, not just what happened to move.
## Guardrail metrics
**Guardrail metrics** protect against unintended harm. These are metrics you don't expect to improve—you just need to ensure they don't get significantly worse.
Think of guardrails as the price of shipping. You might improve checkout completion, but if you also increase refund rates or hurt customer satisfaction, the change isn't worth shipping. You might boost engagement, but if you also increase server costs beyond acceptable limits or degrade app performance, you've created new problems while solving old ones.
Good guardrails measure outcomes that matter strategically but aren't the focus
of your current change. Revenue, retention, and key quality metrics are common
guardrails because they're critical to the business even when they're not what
you're trying to move. Choose guardrails carefully—too many inflate sample size
requirements and delay results, but too few leave you vulnerable to shipping
changes that help one metric while hurting another.
Because you only care about whether a guardrail degrades (not whether it improves), guardrail metrics are tested asymmetrically from success metrics. The [Advancing Experimentation course](../../bootcamp/advancing-experimentation/guardrail-metrics-and-nims) covers inferiority and non-inferiority tests for guardrail metrics in detail.
**Example**
For the checkout simplification experiment, guardrails might include:
**Revenue per completed order**: You're trying to increase completion, not reduce order values. Make sure removing steps didn't make high-value purchases harder.
**Refund rate**: Ensure faster checkout didn't lead to more impulse purchases that customers later regret.
**Customer support contacts**: Check that removing options didn't create confusion requiring support help.
**Site performance metrics**: Verify that code changes didn't slow down the page or introduce errors.
You don't expect checkout changes to improve these metrics, but you need confidence they didn't get worse before shipping.
## Diagnostic metrics
**Diagnostic metrics** answer a simpler question: did the experiment run correctly? Before you interpret any results, you need to know whether the treatment was actually applied, whether users were randomized properly, and whether the implementation worked as intended.
Most platforms track many diagnostic metrics automatically. Sample ratio checks ensure you got the expected number of users in each variant. Exposure metrics confirm users actually saw your treatment. Implementation-specific metrics validate technical details like whether code deployed correctly or whether UI elements rendered.
You should always check diagnostics before diving into results. An imbalanced sample ratio can indicate a randomization bug or biased exposure logging. The most common cause of exposure bias is logging events conditionally on something affected by the treatment, which skews who appears in each variant. Low exposure means fewer users saw your change, reducing your sample size and statistical power—though the effect estimate for exposed users remains accurate. Technical errors might have affected some users but not others, creating misleading patterns in your data.
**Recommendation:** Review diagnostic checks before interpreting your success metrics to ensure your experiment ran as intended. If diagnostics show issues, investigate and fix them before drawing conclusions from your results.
**In Confidence:** In Confidence, diagnostic checks run automatically for every experiment. The randomization is always sound, so a sample ratio imbalance means the exposure logging is biased rather than a randomization issue.
## Exploratory metrics
**Exploratory metrics** help you understand the story behind your results. Of course you will break results down by segments and look for patterns—and of course that will inform how you make product decisions. But it's critical to understand the pitfalls: post-hoc exploration inflates false positive rates, and great product organizations use exploratory findings primarily to inform future experiments and product iterations, not as a direct basis for shipping decisions. This is covered in detail in [Lesson 10: Segment-level analysis](./segment-analysis).
Unlike success metrics, exploratory metrics don't need to be defined before the experiment starts. You can add them during analysis as questions arise. Saw an unexpected drop in your success metric? Add exploratory metrics to investigate possible causes. Saw a surprising win? Add metrics to understand which user segments benefited most or what behaviors drove the improvement.
This flexibility is powerful because you can't anticipate every question you'll want to ask. The data itself often reveals patterns you didn't predict. Exploratory metrics let you dig deeper into those patterns without the rigidity of pre-defined hypotheses.
**Note:** Exploratory metrics do not affect sample size requirements or the ship/no-ship decision, but they are not statistically free. Running many post-hoc analyses inflates the false discovery rate. Treat exploratory findings as hypotheses to confirm in a follow-up experiment, not as the basis for a ship decision.
**Example**
Continuing the checkout flow example, you might add exploratory metrics during analysis to understand the results. If purchase completion increased, you might explore:
- Average time spent in checkout (did speed matter?)
- Cart abandonment at each remaining step (where do users still drop off?)
- Share of purchases using saved payment methods (did removing steps make saved methods more prominent?)
If results were neutral, you might explore:
- Mobile versus desktop completion rates (did the change only help one platform?)
- New versus returning customer completion (did experience level matter?)
- Order values (did you lose high-value shoppers who wanted more options?)
These metrics help you understand what drove the outcome and what to test next.
**In Confidence:** In Confidence, exploratory metrics can be added at any point during analysis. You can also automate exploration using Actions, which can trigger metric calculations, generate reports, or run analyses based on experiment results.
## Complete metric suite
A complete experiment includes all four metric roles, balanced to give you both confidence in your decision and insight into your results.
At Spotify, a typical experiment uses one to three **success metrics** that define winning, three to five **guardrail metrics** for critical business outcomes, three to five **diagnostic metrics** to validate implementation, and five to ten **exploratory metrics** to understand mechanisms and context. These ranges reflect our experience and serve as a useful ballpark—they are not prescriptive rules. The right numbers depend on the scope of your change, your available sample size, how many things could plausibly go wrong, and how much you need to understand about the mechanism behind your results. The [Sample Size Calculation II course](../../bootcamp/sample-size-calculation-ii/multi-metric-decisions) covers how the number of success metrics affects sample size requirements in detail.
**Example**
**Complete metric suite for an e-commerce checkout simplification:**
**Success metric**:
- Purchase completion rate per session in first 7 days
**Guardrail metrics**:
- Revenue per completed order
- 30-day refund rate
- Customer support contact rate
- Payment processing errors
**Diagnostic metrics**:
- Treatment/control sample ratio
- Share of treatment users who saw simplified flow
- Checkout page load times
- JavaScript error rates
**Exploratory metrics**:
- Average checkout time
- Cart abandonment by step
- Mobile versus desktop completion rates
- Share using saved payment methods
- New versus returning customer completion
- Average items per completed order
**In Confidence:** In Confidence, you define metric roles during experiment setup. Success and guardrail metrics form your decision framework and drive sample size requirements. Exploratory metrics do not affect sample size calculations. Learn more in the [Intro to experimentation course](../../bootcamp/intro-course/success-and-guardrail-metrics).
**Quiz: What distinguishes success metrics from exploratory metrics?**
- Success metrics are always more sensitive than exploratory metrics
- Success metrics can be added during analysis, while exploratory metrics must be defined upfront
- Success metrics guide the ship/no-ship decision, while exploratory metrics help understand why results occurred (correct)
**Quiz: Why can exploratory metrics be added during analysis in Confidence?**
- Because they do not affect sample size requirements or decision criteria (correct)
- Because they are less important than success metrics
- Because they are calculated faster than other metric types
**Quiz: What is the main risk of having too many guardrail metrics?**
- It inflates sample size requirements and delays results (correct)
- It makes your experiment harder to understand
- It increases the chance of false positives in your success metrics
**Quiz: You are testing a new product recommendation algorithm. Average order value is a critical business metric that you do not expect to improve. Which role is most appropriate?**
- Success metric, because it is critical to the business
- Exploratory metric, to understand if recommendations affect purchase patterns
- Guardrail metric, to ensure recommendations do not hurt purchasing behavior (correct)
## Notes for nerds
**Risk mitigation and metric types.** One practical way to think about metric roles is through a risk mitigation lens: success metrics confirm you achieved your goal, guardrails protect against harm, and diagnostics confirm your experiment ran correctly. Spotify has written about this framing in the context of running experiments with smaller samples—where the choice of metric roles directly affects how much risk you're taking on at each stage. The [Confidence blog post on experimenting with smaller samples](https://confidence.spotify.com/blog/smaller-sample-experiments) digs into this, including how the risk mitigation ladder connects to metric type selection.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/segment-analysis
# Lesson 10: Segment-level analysis
> When you discover that a metric behaves differently across user groups, it's tempting to solve this by creating filtered metrics for each cohort. Don't. Filtered metrics are statistically inefficient and obscure the aggregate story. Segment analysis gives you the diagnostic power you actually need—without the cost.
## The temptation: a metric for every cohort
Imagine you run an experiment and find that new users respond very differently
from power users. The aggregate result looks muddled. The obvious instinct is to
clean this up at the metric level: define "streams for new users" and "streams
for power users" as separate primary metrics so each can be evaluated cleanly.
This feels rigorous. It is actually a mistake.
When you filter a metric to a subpopulation, you reduce your sample size. A
smaller sample means higher variance in your estimate. Higher variance means
lower statistical power—you need a larger true effect to reliably detect it, and
your confidence intervals get wider. You are not gaining precision; you are
trading away sensitivity.
Worse, you lose the aggregate story entirely. A change that helps new users and
hurts power users will look like a success if you are only watching the new user
metric. The trade-off is invisible.
**Example**
A team notices that a new onboarding flow lifts 7-day streams for new users. They're tempted to make "new user streams" the primary metric for future onboarding experiments. But new users are 15% of the experiment population. Filtering to that 15% means their experiments are now underpowered for the same effect sizes—and they can no longer see whether changes that help new users are hurting everyone else.
## What to do instead: aggregate metric, exploratory segments
The right structure separates two distinct jobs:
**The primary metric** captures the aggregate effect across all users. It has
the full sample size behind it, which maximizes statistical power and captures
trade-offs that would cancel or hide in a filtered view.
**Segment analysis** is how you understand that effect. After you have the
aggregate result, you break it down by pre-specified user groups to diagnose
what is driving it, where the effect is concentrated, and whether any group is
being meaningfully harmed.
This is not a demotion for segment-level thinking—it is a clarification of its
role. Segments are diagnostic tools, not primary outcomes. They answer *why* and
*for whom*, after the primary metric answers *whether*.
## Pre-specify your segments
The risk in exploratory segment analysis is obvious: if you scroll through 20
cuts after seeing the data and report the interesting ones, you will always find
something. At a 5% significance threshold across 20 segments, you should expect
roughly one spurious significant result even when the treatment has no real
effect on anyone.
The fix is to decide which segments you will examine before you look at results.
Pre-specified comparisons carry full statistical weight. Post-hoc ones are leads
to follow up, not conclusions to act on.
The most practical way to do this is a **standard segment set**—a fixed list
your team runs after every experiment. Good candidates are segments where, if
you found a strong effect, it would change what you do:
- **New versus returning users**—onboarding dynamics often differ sharply
- **Free versus premium**—what works for one tier may not work for the other
- **Mobile versus desktop**—platform differences can reflect implementation
quality as much as concept quality
- **Engagement tier**—power users and casual users can respond very differently
**Example**
A streaming platform runs every experiment with a standard breakdown: new users (under 30 days), returning users, free tier, premium tier. A new recommendation algorithm shows a neutral aggregate result. The standard breakdown reveals +8% streams for new users and −3% for premium users—effects that cancel in aggregate. Because these segments were pre-specified, the finding carries full statistical weight and the team knows they have a real trade-off to resolve before shipping.
## The exception: guardrail metrics
There is one legitimate use of cohort-specific metrics: guardrails. A guardrail
metric is not a success criterion—it is a constraint. "Do not hurt new users"
is a reasonable guardrail even if new user streams are not your primary metric.
The distinction matters:
- **Primary metric (filtered)**: bad. Underpowered, loses aggregate signal.
- **Guardrail metric (filtered)**: fine. You are not trying to optimize it, just
checking that you haven't broken something for a specific group.
**Note:** Post-hoc segment findings—ones that emerged from exploring the data rather than
pre-specified analysis—should be treated as hypotheses to confirm in a follow-up
experiment. The most useful thing a surprising post-hoc finding can do is point
you toward the right question for your next experiment, not justify a ship
decision on its own.
**Quiz: Why is filtering your primary metric to a subpopulation (e.g., 'streams for new users only') a problem?**
- It makes the metric harder to explain to stakeholders
- It reduces sample size, increasing variance and lowering statistical power (correct)
- It violates the assumption of random assignment
**Quiz: You run an experiment and notice post-hoc that users in one city show a surprisingly large positive effect. What should you do?**
- Treat the finding as a hypothesis and run a follow-up experiment to confirm it before acting on it (correct)
- Exclude that city from future analyses to avoid bias
- Ship the change and attribute the success to that city
**Quiz: What is the appropriate role for cohort-specific metrics (e.g., 'new user streams') in an experiment?**
- They should replace the aggregate metric when effects are heterogeneous
- They should only be used when the aggregate metric is not statistically significant
- They can serve as guardrail metrics but should not be primary success metrics (correct)
## Notes for nerds
The statistical cost of filtering is a direct consequence of the **variance of
a sample mean**. If your full experiment population has N users, the variance
of your effect estimate scales as 1/N. If you filter to a subpopulation of size
n < N, the variance scales as 1/n—larger by a factor of N/n. To recover the
same power you had with the full population, you need a true effect size that is
√(N/n) times as large.
For a segment that is 20% of your population, you need an effect roughly
**2.2× larger** to detect it with the same reliability as your aggregate metric.
Most real product effects don't come that much stronger in subpopulations than
in aggregate—so you are often just running an underpowered test while thinking
you've done something more targeted.
The multiple comparisons problem compounds this: the familywise error rate for k
independent tests at significance level α is `1 − (1 − α)^k`. At α = 0.05 and
k = 20 segments, that is roughly 0.64—a 64% chance of at least one false
positive even when nothing is real.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/select-metrics
# Lesson 9: Select metrics
> Bring together everything you've learned into a practical framework for selecting metrics. You'll walk through a complete process from hypothesis to metric suite, understand when to make trade-offs, and leave with a systematic approach you can apply to any experiment.
## You're ready to design metrics
You now understand what makes metrics effective. Metrics measure aggregated user
behavior (lesson 1). They play distinct roles in experiments: success, guardrail,
exploratory, diagnostic (lesson 2). Time windows shape when you measure and when
you see results (lesson 3). Good metrics capture the right behavior without
gaming or unintended consequences (lesson 4). The metric hierarchy connects
tactical work to strategic outcomes (lesson 5). Clear names and documentation
ensure interpretability (lesson 6). Effective variance and influenceability
determine whether you can detect changes (lesson 7). Variance reduction techniques reduce effective variance (lesson 8).
The question isn't whether you understand these concepts—you do. The question is
how to apply them systematically when designing an experiment.
## The framework: From hypothesis to metric suite
### Start with your hypothesis
Every metric choice flows from what you expect to change and why. Write your
hypothesis explicitly: **"[Change] will [impact behavior] because
[mechanism]."**
For example: "Adding one-click checkout will increase purchase completion
because it reduces friction for returning customers" or "Personalized homepage
recommendations will increase discovery streams because they surface relevant
content users wouldn't otherwise find."
Your hypothesis identifies what behavior should change (purchase completion,
discovery streams) and why (reduced friction, better relevance). This points
directly to the metrics you need.
### Identify what to measure
Translate your hypothesis into specific, measurable behaviors. Ask: "If my
hypothesis is correct, what would users do differently?"
If reduced friction increases completion, measure purchase completion rate. If
better relevance increases discovery, measure streams from recommendations.
These are your candidate success metrics—they directly capture the behavior your
hypothesis predicts will change.
Then identify what could go wrong. Faster checkout might increase regretted
purchases—measure return rate. Better recommendations might cannibalize
search—measure search engagement. These become guardrail candidates—metrics you
don't expect to improve but must protect.
**Note:** If you can't clearly articulate what behavior should change and what you need to protect, your hypothesis isn't specific enough. Refine it before selecting metrics.
### Assign metric roles
Build a balanced suite by assigning clear roles to each candidate metric.
Your **success metrics** (1-3) define what winning looks like and guide your
ship/no-ship decision. Choose metrics where changes are directly attributable to
your experiment—typically middle-layer metrics (local or proxy) from the metric
hierarchy. They should be movable by your change and sensitive enough to
detect realistic effects.
Your **guardrail metrics** (3-5) protect critical outcomes. Include top-layer
KPIs like revenue and retention, plus any specific risks your change introduces.
These ensure you don't improve one dimension while breaking another.
**Exploratory metrics** (5-10) enable understanding. These help you interpret
why your success metrics moved and discover unexpected patterns. Unlike success
metrics, you can add these during analysis as questions arise.
**Diagnostic metrics** (automated by most platforms) validate
implementation—sample balance, exposure rates, technical health.
### Evaluate metric characteristics
For each success and guardrail metric, validate that it meets your needs across
four dimensions.
First, check **feasibility**—can you compute this reliably with available data?
Do you have sufficient history to estimate variance reduction potential? If not,
you may need to invest in logging infrastructure before proceeding.
Second, assess **effective variance and influenceability**, which together
determine whether you'll detect changes. You need both low effective variance
(considering temporal correlation and regression adjustment, not just raw
variance) and high influenceability (your change must actually move the metric).
Remember that binary metrics aren't automatically more sensitive—they reduce
both variance and influenceability. A continuous metric with strong temporal
correlation often outperforms a binary metric after applying variance reduction.
Third, ensure **interpretability**—does the name clearly communicate what's
being measured (what behavior, for whom, over what time period)? Will
stakeholders understand what it means when the metric moves?
Finally, verify **alignment**—does this metric connect to what you actually care
about? If you're using a proxy metric, have you validated that it correlates
with the ultimate outcome?
**Recommendation:** If a metric fails feasibility, invest in the logging infrastructure you need—don't compromise by measuring the wrong thing. If it fails variance or influenceability tests, refine the metric or reconsider your experimental approach.
### Configure time windows
Time configuration determines when you measure and when you see results. For
each metric, you need to make three choices.
**When does measurement start?** The exposure offset controls this. Use 0 days
for immediate effects, or 7+ days to measure behavior after novelty effects
fade. Consider using multiple metrics with different offsets to understand both
short-term response and sustained impact.
**How long do you measure each user?** The aggregation window determines this.
Match the window to the natural cycle of your user behavior and how long effects
take to manifest.
**When do users appear in results?** You have two options. Closed windows
(measuring at the end of the window) give cleaner interpretation—all users
measured over exactly the same period—but results appear slower. Use these for
success metrics guiding final decisions. Cumulative windows give faster results by including users as soon as they enter the
window, even if incomplete. Use these when you need earlier monitoring, such as
for guardrails or when monitoring success metrics before they mature.
### Document everything
Write down for each metric: the specific behavior being measured, who it applies
to (all users, activated users, exposed subset), the time configuration,
expected direction of change, and why you chose this metric. Include calculation
logic, filters, and any caveats.
This documentation serves three purposes: it ensures everyone interprets results
consistently, it helps you catch issues before launching, and it creates
a learning artifact for future teams.
## Trade-offs
Real constraints force trade-offs. You can't always have perfect alignment,
ideal sensitivity, and fast results simultaneously. Here's how to navigate
common tensions.
When **sensitivity conflicts with business alignment**, remember that
business-critical metrics like revenue and retention often have high variance or
require slow measurement. Use sensitive proxies as primary success metrics while
monitoring business metrics as guardrails. Just validate that improvements in
your sensitive metric actually translate to business outcomes.
When **feasibility limits ideal measurement**, proxy metrics become necessary.
Use them when direct metrics aren't available, but validate the correlation with
historical data. Document the relationship clearly so others understand what the
proxy represents.
When **time constraints conflict with clean interpretation**, semi-open windows
give faster monitoring at the cost of interpretation complexity. Use them for
early signals, but plan to validate with closed windows before making
irreversible decisions.
The key is making these trade-offs deliberately, documenting them, and
understanding their implications.
## A complete example
**Example**
**Context**: An e-commerce platform tests a visual search feature that lets users upload photos to find similar products.
**Hypothesis**: "Visual search will increase purchase conversion for fashion categories because it helps users find specific styles when they can't describe them in text."
**Metric design**:
**Success metrics**:
1. **Purchase conversion rate for visual search users in fashion categories** (cumulative)
- *Why*: Directly measures the hypothesized behavior change
- *Characteristics*: Highly influenceable (scoped to visual search users), moderate variance, clear interpretation
2. **Visual search usage rate among fashion shoppers** (cumulative)
- *Why*: Validates feature adoption—if users don't use it, conversion can't improve
- *Characteristics*: Very sensitive, fast signal, enables understanding of null results
**Guardrail metrics**:
1. **Overall revenue per user** (cumulative)
- *Why*: Ensure visual search doesn't cannibalize other purchase paths
2. **Text search usage rate** (cumulative)
- *Why*: Detect if visual search replaces rather than complements text search
3. **Mobile app performance scores** (cumulative)
- *Why*: Visual search uses image processing—ensure it doesn't degrade experience
**Exploratory metrics** (can be added during analysis):
- Category distribution of visual search queries
- Success rate of visual searches (found similar products)
- Average product price from visual vs text search
- Cart add rate from visual search results
- Share of searches that are visual vs text
**Trade-offs made**:
- Using "purchase conversion for visual search users" rather than "overall site conversion" (more influenceable but narrower scope—monitoring overall conversion as guardrail)
- Using cumulative metrics without windows since e-commerce users (especially anonymous/cookie-based) may not return, making fixed windows less appropriate
## You've got this
You now have a systematic approach for selecting metrics. You understand how to
start with your hypothesis, translate it into measurable behaviors, assign
appropriate roles, validate characteristics, configure time windows, and make
deliberate trade-offs when necessary.
The framework isn't rigid—adapt it to your context. But the core principles
remain: be explicit about what you expect to change and why, measure the right
behavior even if it's harder, protect critical outcomes, and document your
choices so others can learn from them.
Metric design is iterative. Your first attempt won't be perfect. You'll learn
from each experiment what worked, what didn't, and how to refine your
measurement approach. That's expected and valuable—each iteration builds your
intuition and your organization's measurement capabilities.
**Recommendation:** Start applying this framework on your next experiment. Walk through each step, document your decisions, and review the metric choices with a colleague before launching. After the experiment concludes, reflect on what you learned about metric design and how you'd approach it differently next time.
**Quiz: What should drive your metric choices in an experiment?**
- The metrics that are easiest to calculate
- The metrics your team used in the last experiment
- Your explicit hypothesis about what will change and why (correct)
**Quiz: Why might you choose a proxy metric with moderate variance over a business metric with high variance?**
- The proxy metric may have higher influenceability and faster measurement while still connecting to business outcomes (correct)
- Business metrics should never be used in experiments
- Proxy metrics are always more accurate
**Quiz: When should you use closed windows versus semi-open windows?**
- Always use closed windows because they are more accurate
- Use closed windows when clean interpretation matters most; use semi-open when you need earlier monitoring (correct)
- Use semi-open windows for all metrics to get faster results
**Quiz: What is the most important step when using a proxy metric instead of the direct business outcome?**
- Validate that the proxy correlates with the outcome you care about (correct)
- Ensure the proxy is easier to calculate than the direct metric
- Get approval from leadership before using proxies
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/strategic-metrics
# Lesson 5: Strategic metrics
> In this lesson, you learn about the strategic metric hierarchy and how different types of metrics serve different purposes. You explore KPIs, composite metrics, proxy metrics, and local metrics, and understand how they connect to business goals.
## The metric hierarchy
Not all metrics serve the same purpose. Some represent the most important measures of business health. Others measure specific features or user flows. Understanding this hierarchy helps you choose the right metrics for different decisions and communicate effectively across the organization.
Think of metrics as forming a three-layer pyramid. At the top sit your strategic measures: the KPIs that define organizational success. In the middle are proxy and local metrics that measure team and product area impact. At the base are feature and functionality metrics for day-to-day optimization. Each layer connects to the one above it, creating a path from tactical improvements to strategic outcomes.
**Note:** This pyramid is one of many frameworks for structuring metrics—others include HEART, AARRR (Pirate Metrics), OKRs, and more. Use whatever fits your organization. The underlying principle is always the same: every metric should connect, directly or indirectly, to something your business ultimately cares about.
## Key Performance Indicators (KPIs)
At the top of the pyramid are **Key Performance Indicators**—the metrics that
define success for your organization or product. These are the "North Star"
metrics that senior leadership tracks and that drive strategic decisions.
A good KPI is directly tied to business objectives and reflects meaningful user
or business value. It's stable enough to track over time, understandable by all
stakeholders, and actionable—teams can actually influence it. Monthly active
users, subscription conversion rate, and revenue per customer are common KPIs
because they meet these criteria.
**Example**
**KPIs across industries:**
A streaming service tracks monthly active users, paid subscribers, and listening
hours per user. An e-commerce platform measures gross merchandise value,
customer acquisition cost, and average order value. A SaaS company focuses on
monthly recurring revenue, net revenue retention, and customer lifetime value. A
gaming company monitors daily active users, average revenue per user, and player
retention.
These metrics represent fundamentally different businesses, but they share the same characteristics: they're tied to business success, leadership pays attention to them, and teams can influence them through product decisions.
**Note:** In experiments, KPIs most commonly serve as guardrail metrics to ensure product changes don't harm core business outcomes. Less frequently, they're used as success metrics when a change is expected to directly impact these high-level measures.
## Composite metrics
**Composite metrics** combine multiple individual metrics into a single measure.
They're tempting when success requires improving multiple dimensions
simultaneously—you can simplify reporting by reducing many metrics to one
number, capture multidimensional success, and reduce multiple testing problems
in experiments.
But composites come with serious trade-offs. When a composite metric moves, it's
hard to interpret which component drove the change. They can hide important
trade-offs between components—one metric improving while another degrades, with
the composite showing a neutral result. And they require careful design and
validation to ensure the weighting actually reflects what matters.
**Recommendation:** Avoid composite metrics. When you do use them, always monitor
the individual components so you can understand what's driving changes in the
composite.
## The middle layer: Proxy and local metrics
Between KPIs at the top and daily operations at the bottom sits the middle
layer: **proxy and local metrics**. These measure team and product area impact.
**Local metrics** measure behavior in specific areas: checkout completion rate,
dashboard usage, or level completion rate. These help you understand how
particular product areas perform and give teams clear signals about their
impact.
**Proxy metrics** fit here because they connect tactical work to strategic
outcomes. Week 1 retention proxies for long-term retention. Add-to-cart rate
proxies for purchase intent. These metrics bridge the gap between what teams can
directly influence and what the business ultimately cares about.
Proxy metrics are often genuinely necessary—no team can run a six-month experiment to directly observe long-term retention. Their value lies in making outcomes measurable on an experimentally practical timescale. Always validate proxies with historical data—a proxy that seems logical but doesn't actually correlate with the real outcome will lead you to optimize for the wrong thing. Revisit that validation as your product and user base evolve, since the relationship between a proxy and its underlying outcome can drift over time.
## The bottom layer: Feature and functionality metrics
At the base of the pyramid are **feature and functionality metrics**, the
day-to-day optimization tools.
**Feature metrics** measure interaction with specific features: wish list
additions, collaboration invites sent, or power-up purchases. These tell you
whether users are adopting and using individual capabilities.
**Functionality metrics** describe whether features work as intended from a
technical perspective—page load times, search response times, API error rates.
These are the quality assurance and performance monitoring tools.
The three layers connect: improvements in feature and functionality metrics
should drive proxy and local metrics, which should ultimately contribute to your
KPIs.
## How layers connect
The power of this hierarchy comes from understanding how the three layers
connect. Improvements at the bottom should cascade upward, ultimately
contributing to your strategic measures at the top.
**Example**
**How improvements cascade through the pyramid:**
**Bottom layer**: You reduce page load time from 2 seconds to 0.5 seconds
(functionality metric) and see increased feature usage as users engage with the
faster experience (feature metric).
**Middle layer**: The improved feature usage drives higher checkout completion
rates in that product area (local metric). This becomes a validated signal that
faster experiences improve conversion (proxy metric).
**Top layer**: The improved conversion rates across product areas contribute to
increased gross merchandise value—a key strategic measure (KPI).
Each improvement connects to the layer above it, creating a chain from daily optimization to strategic business impact. In practice, these causal chains are rarely this clean—faster page load, for instance, may attract both high- and low-intent users, and the net effect on GMV depends on the composition of that additional traffic. Treat the hierarchy as a reasoning framework for building and testing hypotheses, not as a guarantee that bottom-layer improvements will propagate upward.
## The right layer for your question
The layer of metric you choose depends on what question you're trying to answer.
For strategic planning, use the top layer—KPIs that set organizational direction
and goals. For team road maps, use the middle layer—proxy and local metrics that
measure team impact on the business. For feature development and quality
assurance, use the bottom layer—feature and functionality metrics for day-to-day
optimization.
In experimentation, you typically mix layers. Your success metric is often from
the middle layer (a local or proxy metric)—specific enough to detect the
change's impact, but meaningful enough to matter. Your guardrail metrics
typically come from the top layer (KPIs), ensuring your optimization doesn't
harm core business outcomes.
**In Confidence:** In Confidence, required metrics are automatically added to all experiments on a surface and checked continuously for deterioration. Add Strategic KPIs and general app quality metrics as required metrics on the global surface to ensure all experiments check for deterioration of these metrics. Add more local metrics as required metrics on associated surfaces.
**Quiz: What distinguishes a KPI from a local metric?**
- KPIs are always more accurate than local metrics
- KPIs are tied to organizational business objectives while local metrics measure specific product areas (correct)
- KPIs are calculated daily while local metrics are calculated weekly
**Quiz: Why should you monitor individual components when using composite metrics?**
- Individual components are more accurate than composites
- To understand which specific behavior drove changes and detect hidden trade-offs (correct)
- Composite metrics are deprecated and should not be used
**Quiz: What is the most important validation step when choosing a proxy metric?**
- Ensure the proxy is easier to measure than the direct outcome
- Verify with historical data that the proxy actually correlates with the outcome you care about (correct)
- Confirm that leadership approves of the proxy metric
**Quiz: In experiments, which pyramid layers typically provide success metrics versus guardrail metrics?**
- Success: top layer (KPIs); Guardrails: bottom layer (feature/functionality)
- Success: bottom layer (feature/functionality); Guardrails: middle layer (proxy/local)
- Success: middle layer (proxy/local); Guardrails: top layer (KPIs) (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/time-considerations
# Lesson 3: Time considerations
> In this lesson, you learn how time windows shape metric behavior and interpretation in experiments. You start with the simplest cumulative approach, then explore how to configure exposure offsets and aggregation windows, understand the differences between closed and cumulative windows, and learn how to choose the right time configuration for your use case.
## Why time windows matter in metrics
Every metric aggregates measurements over some time window. The way you configure this time window fundamentally affects what you're measuring and when you can interpret your results. Consider a simple metric like "streams per user" in the first week after signup. This seemingly straightforward metric raises several questions: Does "first week" start immediately at signup or after some delay? Should we wait until all users complete their full week before including them in results, or include them as they progress through the week?
The answers to these questions determine not just what you measure, but how
quickly you get results and how easy those results are to interpret. A metric
that waits for all users to complete their measurement window gives you the
cleanest interpretation but delays your results. A metric that includes users as
soon as they're exposed gives you faster feedback but requires more careful
interpretation.
Watch this video for an overview of the different ways to handle time in metrics
in 4 minutes and 34 seconds.
[Video: https://www.youtube.com/watch?v=3faa2_gFjWQ]
## The simplest approach: cumulative metrics
The most straightforward way to configure a metric is to include all measurements from all exposed entities as soon as they're exposed, with no upper limit on the time period. This **cumulative approach with no window** starts counting immediately at exposure and continues indefinitely.
With this approach, a user exposed on day 1 has 10 days of measurements on day 10, while a user exposed on day 9 has only 1 day of measurements. This means different users are always measured over different time periods—you're comparing users who have had very different amounts of time to exhibit the behavior you're measuring.
**Note:** In cumulative metrics without a window, some entities are always measured over longer periods than others because not all entities are exposed at the same time. This makes these metrics hard to interpret. Only use these metrics when entities are short-lived.
This approach is primarily useful when you're working with short-lived entities
like cookie-based users who may not return for multiple sessions. In these
cases, trying to follow the same entity over a multi-day window doesn't make
sense because the entity itself may not exist that long.
**Example**
An e-commerce site runs experiments on anonymous visitors identified by cookies. Many visitors never return after their first session. A metric measuring "average order value" configured with no window makes sense here because:
- Following individual cookies over multiple days often means measuring nothing (they don't return)
- The business cares about the total order value from all visitors, regardless of when they were exposed
- The short-lived nature of the entity makes time windows less meaningful
For persistent entities like logged-in users, this approach creates problems. Users exposed early in the experiment always have more time to contribute measurements than users exposed later, making fair comparisons difficult. This is where time windows become essential.
## How time windows work
Time windows solve the problem of unequal measurement periods by defining a fixed time period for each user relative to their exposure. Window-based metrics use two key parameters: the **exposure offset** and the **aggregation window**.
**In Confidence:** In Confidence, you configure the exposure offset and aggregation window when defining a metric.
The **exposure offset** defines how long to wait after a user's exposure before starting to collect their measurements. Think of it as a waiting period. If you set an offset of 7 days, the first week of behavior is ignored and measurements only start from day 8 onward.
The **aggregation window** defines the length of time over which you collect measurements for each user. A 7-day aggregation window means you're collecting data for exactly 7 days (relative to each user's exposure time plus any offset). The window is always relative to when each individual user was exposed, not to calendar time.
The figure illustrates how time windows work to aggregate values within entities. The horizontal bar shows when the user was first exposed to the experiment. The dashed part of the box is the exposure offset, and the solid box is the aggregation window. The metric calculation includes the measurements that fall within the aggregation window.
### When to use exposure offset: novelty and change-aversion effects
Sometimes the effect you want to measure isn't the immediate response to a change but how users behave after they've had time to adapt to it. This is where exposure offset becomes valuable.
Two opposing time-dependent effects can distort early measurements. The first is **novelty effects**: users encountering something new temporarily change their behavior—a redesigned interface might see increased engagement simply because users are exploring the change, even if the long-term effect is neutral or negative.
The second is **change aversion** (also called primacy effects in the experimentation literature): users accustomed to the old experience may initially resist or underuse the new one, suppressing the apparent effect until they adapt. Novelty and change aversion are among the most common sources of short-term metric distortion in online experiments.
These two forces work in opposite directions. Novelty inflates early results; change aversion suppresses them. Either can mislead you about the true long-term impact of a change.
**Example**
A music streaming service redesigns their playlist creation flow. They configure two metrics:
- **Immediate response**: 0-day offset, 7-day window—measures first week behavior
- **Sustained impact**: 7-day offset, 7-day window—measures second week behavior after the initial adjustment period
The immediate response metric shows a 15% increase in playlist creation. The sustained impact metric shows only a 3% increase. This suggests most of the initial lift was novelty. Had the redesign created friction for habitual users, the immediate metric might instead have shown a decrease that recovered as users adapted—an example of change aversion.
Using an offset lets you deliberately measure behavior after the initial adjustment period. This is particularly useful when your hypothesis is about sustained behavior change rather than immediate response.
**Recommendation:** When you suspect novelty or change-aversion effects, configure multiple metrics with different offsets: one capturing immediate response and one capturing sustained behavior after the initial period. Comparing the two gives you a more complete picture of your change's true impact.
## Two ways to use windows
Now that you understand what a window is—a fixed time period for each user defined by offset and duration—the question becomes: when should users appear in your metric results? Two approaches exist for window-based metrics, each representing a different trade-off between interpretability and speed of results.
### Closed windows: at the end of a window
Metrics that include entities **at the end of a window** wait until each user completes their full measurement window before including them in the results. This is the most rigorous approach because every user included in the metric results has been measured over exactly the same time period relative to their exposure.
The illustration shows how this works over time. On any given day of your experiment, the metric results include only users who have completed their full aggregation window. Users exposed on day 1 appear in the results after `exposure offset + aggregation window` days. Users exposed on day 2 appear one day later, and so on.
This approach has an important implication for when you see results. If you create a metric measuring behavior during the second week after exposure (offset of 7 days, window of 7 days), you won't see any results until 14 days after launch. Before that, no user has completed their second week yet. It's also worth noting that closed windows only include users exposed early enough to complete the full window—users exposed near the end of the experiment are excluded. For long windows (30+ days), this means your results increasingly reflect earlier-exposed users, which can introduce bias if user composition shifts over the course of the experiment.
**Example**
A streaming platform tests a new recommendation algorithm. They create a metric measuring "streams in the second week after exposure" configured as:
- **Exposure offset**: 7 days
- **Aggregation window**: 7 days
- **Include users**: At the end of a window
This metric shows no data for the first 13 days. On day 14, the first exposed users complete their second week and appear in the results. Each subsequent day adds more users who have completed their full second week.
### Cumulative windows: include users during a window
Metrics that include entities **cumulatively during a window** include users as soon as they enter their measurement window, even if they haven't completed it yet. This gives you earlier results but requires more careful interpretation because different users in your metric results have been measured over different amounts of time.
With this approach, you start seeing results as soon as the first users reach the start of the window (after the exposure offset). A user exposed on day 1 appears in the results on day 8 (after a 7-day offset), even though they've only been measured for one day of their 7-day window. The next day, that same user has two days of data in the window, and so on.
This means the metric results change their meaning over time. Early in the experiment, the metric represents an average of users measured over different fractions of the full window. Only after `exposure offset + aggregation window` days do all users in the results have their full window measured.
**Example**
Using the same recommendation algorithm example with cumulative inclusion:
- **Exposure offset**: 7 days
- **Aggregation window**: 7 days
- **Include users**: Cumulatively during a window
This metric shows first results on day 8, when the earliest exposed users enter their second week. On day 8, these users have only 1 day of second-week data. On day 9, they have 2 days of data. By day 14, the earliest users have their full 7 days of second-week data, but newer users still have partial windows.
## The right time configuration
The decision of which time configuration to use follows a logical progression: first consider your entity type, then decide whether to use windows, and finally choose how to include users in results.
### Step 1: consider your entity type
**For short-lived entities** like cookies or anonymous sessions that rarely return, use **cumulative metrics with no window**. Time windows are impractical when entities don't persist long enough to complete them. Accept that different measurement periods are unavoidable in this case.
**For persistent entities** like logged-in users, accounts, or devices, use **window-based metrics** for fair comparisons. Windows ensure each user is measured over the same time period, regardless of when they were exposed. Continue to step 2 to choose your window approach.
### Step 2: choose your window approach (for persistent entities)
| Approach | When to Use | Trade-off |
|----------|-------------|-----------|
| **Closed windows**
(at the end of window) | Interpretability is paramount and you can wait for complete data. Best for success metrics when making final decisions. | Slower results, but cleanest interpretation—every user measured over exactly the same period. |
| **Cumulative windows**
(during window) | You need earlier results for monitoring. Common for guardrail metrics or when you want to monitor success metrics before they reach full maturity. | Faster results, but more complex interpretation—users at different stages of their windows. |
### Step 3: configure offset and window duration
**Exposure offset** controls when measurement starts. Use 0 days to measure immediate response, or 7+ days to skip novelty or change-aversion effects and measure sustained behavior. Consider using multiple metrics with different offsets to understand both short-term and long-term impact.
**Window duration** should match the natural cycle of your user behavior and how long it takes for the effect to manifest. Make sure your window is compatible with your fact data frequency—don't use hourly windows if your facts are generated daily.
**Quiz: What does the exposure offset parameter control in a metric?**
- How long to wait after exposure before starting to collect measurements (correct)
- How long to collect measurements for each user
- How many users to include in the experiment
**Quiz: A metric configured to include users 'at the end of a window' with a 7-day offset and 7-day window will show first results on which day after experiment launch?**
- Day 7
- Day 8
- Day 14 (correct)
**Quiz: When should you use metrics with no window (cumulative without window)?**
- For logged-in users to track long-term behavior
- Always, because they give the fastest results
- Only for short-lived entities where time windows are impractical (correct)
**Quiz: What is the main trade-off between 'at the end of a window' and 'cumulatively during a window'?**
- Interpretability versus speed of results (correct)
- Cost versus accuracy
- Sample size versus statistical power
## Notes for nerds
Cumulative metrics are tricky both from an interpretation perspective and an
inference perspective. Spotify's engineering team has dug deep into both topics:
* [Bringing sequential testing to experiments with longitudinal data part 1: the peeking problem](https://engineering.atspotify.com/2023/7/bringing-sequential-testing-to-experiments-with-longitudinal-data-part-1-the-peeking-problem-2-0)
* [Bringing sequential testing to experiments with longitudinal data part 2: the peeking problem](https://engineering.atspotify.com/2023/7/bringing-sequential-testing-to-experiments-with-longitudinal-data-part-2-sequential-testing)
* [It's About Time: What A/B Test Metrics Estimate](https://arxiv.org/html/2411.06150v1)
**Survivorship bias in cookie-based experiments.** The recommendation to use cumulative no-window metrics for short-lived cookie entities has a hidden cost: users who don't return after their first session are excluded from your analysis entirely. These non-returning users are precisely those most likely to have had a poor experience. By measuring only returning users, your metric reflects a survivor population—the happy path—rather than the full distribution of user outcomes. Keep this in mind when interpreting results from anonymous-visitor experiments.
**Variance non-stationarity.** Metric variance is not constant over the life of an experiment. In the early days, variance is typically elevated—novelty effects, an evolving mix of new versus returning users, and exploratory user behavior all contribute to higher noise. Variance tends to stabilize as the experiment matures. This matters in practice: sample size estimates based on historical variance may be optimistic for early experiment phases, and sequential testing methods that assume fixed variance need to account for this non-stationarity when applied to longitudinal data.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/variance-reduction
# Lesson 8: Variance reduction
> Understanding variance reduction is essential for metric selection, not just statistics. A metric with high raw variance might still be the best choice if it has a strongly correlated covariate—because variance reduction can bring its effective variance well below that of a seemingly simpler metric. This lesson explains how regression adjustment works, what drives how much reduction you get, and how to handle outliers so you can make smarter metric choices.
## Why metric selection and variance reduction are inseparable
When you choose a metric, raw variance is only half the picture. What matters for your experiment's power is **effective variance**—variance after applying regression adjustment. A continuous metric like total streams per user may look noisy in isolation, but if user behavior is stable over time, a pre-experiment covariate will absorb most of that noise. The result can be a far more sensitive metric than a binary alternative that seemed cleaner on the surface.
This means you can't evaluate metrics without understanding variance reduction, and you can't apply variance reduction thoughtfully without understanding which metrics it works well for. The two decisions are made together.
## CUPED, CUPAC, and their relatives are all regression adjustment
CUPED, CUPAC, and every other branded variance reduction technique in online experimentation are fundamentally the same thing: **regression adjustment**. You regress a pre-experiment covariate out of the outcome and analyze the residuals. This reduces variance by the factor (1 − ρ²), where ρ is the correlation between the covariate and the outcome.
The statistical principle goes back decades. The efficiency gains from adjusting for pre-treatment covariates were formalized by Cochran (1957) under the name **analysis of covariance (ANCOVA)**, building on the potential outcomes framework introduced by Neyman (1923). What the 2013 CUPED paper genuinely contributed was adapting these classical results to online A/B testing at scale, extending them to ratio metrics, and adding a conceptually important insight: by estimating the adjustment coefficient from *pre-experiment data* rather than the experimental sample, the adjusted outcome is unbiased without requiring any modeling assumption about the covariate-outcome relationship. Because the pre-experiment period cannot be influenced by treatment, the adjustment is valid by design.
CUPAC, for instance, is CUPED with an ML-predicted outcome as the covariate instead of the raw pre-experiment metric—useful when the simple covariate is weakly correlated. Most subsequent methods follow the same pattern: a different covariate choice within the same regression framework. When you see a new acronym, the question that cuts through is: "what covariate, and how correlated is it with the outcome?" Same principle, different covariate.
**Note:** The (1 − ρ²) factor is the key lever. If your covariate has a correlation of 0.7 with the outcome, you reduce variance by 51%. A correlation of 0.9 gives you 81% variance reduction. This is why choosing a strongly correlated covariate matters much more than which specific method you use.
## How much variance reduction should you expect?
The answer depends on how stable the metric is for your users over time—specifically, how well past behavior predicts future behavior.
At Spotify, for behavioral metrics with high temporal correlation—such as listening minutes or streams per user—variance reduction of 50-80% is common.
For sparser metrics like purchase conversion or binary activation outcomes, reductions of 20-30% are more typical. Your results will depend on how stable the metric is for your specific user base and time horizon.
**Example**
**High temporal correlation—strong variance reduction:**
A user who streamed 400 minutes last week is very likely to stream a similar amount next week. Using last week's streaming minutes as the CUPED covariate gives a high ρ, which translates to large variance reduction. The experiment reaches the same statistical power in substantially less time—or detects a smaller effect with the same sample.
**Low temporal correlation—modest variance reduction:**
Whether a user converts to a paid plan this week tells you relatively little about whether they'll convert again next week (most users either have or haven't converted). The covariate has low predictive power, ρ is small, and the variance reduction is correspondingly modest.
## The best covariate: the metric itself
In practice, the single most reliable covariate for most behavioral metrics is the pre-experiment measurement of the metric you're trying to reduce variance on. If you're measuring "streams per user" in the experiment, using "streams per user in the weeks before the experiment" as your covariate tends to be hard to beat.
The intuition is straightforward: past behavior is the best predictor of future behavior. A user's pre-experiment streaming behavior reflects their baseline preferences, habits, and engagement level far better than any demographic or derived feature. This gives a high ρ, which translates directly into large variance reduction.
It turns out this simple choice—the pre-experiment metric itself, exactly what CUPED uses—is hard to beat. Even with sophisticated feature engineering or ML-predicted outcomes, the extra variance reduction you can squeeze out beyond it is limited: at most a further 29% narrowing of confidence intervals [(Ting and Hung, 2023)](https://arxiv.org/abs/2311.17858).
More complex covariates can still be worth exploring, but this is a strong default that requires no feature engineering and is easy to explain and audit.
**Recommendation:** When in doubt, start with the pre-experiment measurement of your metric as the covariate. It requires no feature engineering, is easy to explain and audit, and performs well empirically. Move to more complex covariates only if there's a specific reason to expect they'll do better.
## Outlier treatment: cap versus winsorize
Variance reduction via regression adjustment addresses noise from natural behavioral variation. But a separate problem is outliers: a small number of extreme users can dominate metric variance and distort your estimates even after regression adjustment.
Two common approaches exist for handling this.
### Cap
**Capping** sets an absolute maximum value for the metric. For example, you might cap daily streams at 500. Any user who streamed more than 500 times in a day is treated as though they streamed exactly 500.
The advantage of capping is that the threshold is fixed, predictable, and consistent across experiments. If your team agrees that 500 streams per day is the cap, every experiment that uses this metric applies the same rule, regardless of the population being tested or when the experiment runs.
### Winsorize
**Winsorizing** is conceptually similar but uses a percentile-derived threshold rather than a fixed value. You might winsorize at the ninety-ninth percentile, replacing any value above that percentile with the percentile value itself.
The problem with winsorizing is that the threshold is a function of the sample. Different experiments targeting different user populations will produce different capping points—and those differences are unpredictable and non-comparable. An experiment targeting heavy users might winsorize at 900 streams per day; one targeting casual users might winsorize at 80. These are not the same metric, even if the winsorizing rule is nominally identical.
**Recommendation:** Prefer capping over winsorizing for metrics used consistently across experiments. An absolute cap is predictable, stable, and easy to reason about when comparing results across teams and over time. Reserve winsorizing for exploratory analysis where cross-experiment comparability is not a concern.
Both approaches involve a trade-off: you lose some information about extreme users in exchange for lower variance and more reliable estimates. The key question is whether the extreme values reflect genuine user behavior you want to capture, or noise and edge cases you'd rather control for. In most experimentation contexts, the latter is more common—an extreme outlier is rarely the user your feature change is targeting.
**Quiz: What do CUPED, CUPAC, and similar variance reduction techniques have in common?**
- They all use machine learning models to predict outcomes
- They are all fundamentally regression adjustment—regressing a pre-experiment covariate out of the outcome (correct)
- They are only applicable to ratio metrics
**Quiz: If the correlation between your covariate and outcome is ρ = 0.8, by approximately what factor does regression adjustment reduce variance?**
- 20% reduction—variance is multiplied by ρ = 0.8
- 36% reduction—variance is multiplied by ρ² = 0.64
- 64% reduction—variance is multiplied by (1 − ρ²) = 0.36 (correct)
**Quiz: Why is capping generally preferred over winsorizing for metrics used across multiple experiments?**
- An absolute cap is predictable and consistent across experiments; winsorizing produces different thresholds for different populations, making results non-comparable (correct)
- Winsorizing is more computationally expensive
- Capping removes more outliers than winsorizing
## Notes for nerds
**Variance reduction for ratio metrics.** In Confidence, variance reduction is applied to all metric types—including ratio metrics—by using the method introduced in [Ying Jin and Shan Ba (2021)](https://arxiv.org/abs/2110.13406), which extends regression adjustment to ratio metrics directly. This means you don't need to pre-aggregate ratio metrics to the user level before applying variance reduction; the platform handles the joint estimation of numerator and denominator covariates automatically. For reference, the delta method variance formula and the general framework for ratio metrics in online experimentation are covered in: Deng, A., Lu, J., & Wang, S. (2018). "Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas." *Proceedings of KDD 2018*.
**Covariate selection and rerandomization.** [Schultzberg and Johansson (2020)](https://www.tandfonline.com/doi/full/10.1080/10618600.2020.1753531) examines using historical data to predict experimental outcomes and using those predicted outcomes as covariates—the same covariate construction idea as CUPAC, applied in a rerandomization context. A related result from [Li and Ding (2020)](https://arxiv.org/abs/1906.11291) shows that Mahalanobis-distance rerandomization is asymptotically equivalent to regression adjustment using the same covariates. Together, these papers establish a clean theoretical bridge between design-based variance reduction (rerandomization) and analysis-based variance reduction (CUPED/ANCOVA): the two approaches converge when they use the same covariates.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/what-is-a-metric
# Lesson 1: What is a metric?
> In this lesson, you learn the fundamental definition of a metric and how metrics are created from individual events and measurements. You also explore the many ways metrics are used across organizations to drive product decisions, from performance tracking to experimentation.
## Why metrics matter
Every day, product teams make decisions that affect millions of users. Should you launch that new feature? Is the redesign working? Which version of the experience is better? These questions can't be answered with intuition alone. You need evidence, and that evidence comes from metrics.
Metrics transform raw user behavior into actionable insights. They tell you whether your product is healthy, whether your changes are working, and where to focus your efforts.
The challenge isn't just measuring things; it's measuring the right things in the right way. This course teaches you how to think about metrics: what they are, how to choose them, and how to use them effectively.
## From events and measurements to metrics
At its core, a metric is a number that represents a specific aspect of a system you want to observe and understand. More formally, **a metric is an aggregation across users or sessions** that provides insight into user behavior, product performance, or business health.
To build metrics, you start with raw data about what users do. This data comes in two forms: **events** and **measurements**. An event captures that something happened: a song was streamed, a button was clicked, a purchase was completed. A measurement captures a quantity with a scale or unit: 3.5 minutes of audio consumed, $47.99 in order value, 1,250 bytes downloaded.
Imagine you want to understand how much users engage with Spotify. When someone plays a song, that's an event. The minutes they spent listening is a measurement. User A streamed a song at 9:00 AM (event) and listened for 3.2 minutes (measurement). User B streamed a song at 9:15 AM (event) and listened for 4.7 minutes (measurement). User A streamed another song at 9:30 AM (event) and listened for 2.8 minutes (measurement).
These individual events and measurements don't tell you much on their own. You can't look at millions of stream events and understand engagement patterns. You need to aggregate them across users into something more meaningful.
That's where metrics come in. When you aggregate events—counting streams per user—you create metrics about frequency. When you aggregate measurements—summing listening minutes per user—you create metrics about quantity. "Average streams per user" tells you about typical engagement levels. "Total minutes listened per user" tells you about consumption depth. "Share of users who streamed daily" tells you about habit formation.
The aggregation step across users or sessions is what transforms raw data into insight. Individual events and measurements are just data points. Metrics are the lens through which you understand those data points and make decisions based on them.
**Note:** The key distinction: Events capture that something happened. Measurements capture quantities with scale. Metrics are aggregations of events or measurements across users that provide actionable insights.
## Metric types
Three metric types come up repeatedly in experimentation, and understanding the differences shapes how you measure, analyze, and interpret your data.
**Continuous metrics** measure quantities that can vary across a wide range. When you count streams per user, sum up total listening time, or calculate average session length, you're working with continuous metrics. These metrics capture "how much" or "how many"—they answer questions about quantity, frequency, and magnitude.
The power of continuous metrics lies in their granularity. They don't just tell you whether something happened—they tell you the degree to which it happened. A user who streams 100 songs per week is having a very different experience from a user who streams 5, and continuous metrics capture that difference.
**Example**
**Continuous metrics in practice:**
For a streaming platform like Spotify, continuous metrics might include total streams per user over the past week, or minutes of content played per day. Each user contributes a number that could range from zero to hundreds or thousands.
For an e-commerce site, you might measure average order value or items purchased per month. One customer might buy a single low-cost item while another makes large, multi-item purchases—the continuous metric captures that full range.
For a SaaS product, you might track features used per session or API calls per customer, revealing how deeply different users engage with your platform.
**Binary metrics** take a different approach. Instead of measuring how much, they measure whether. Did the user stream this week or not? Did they complete a purchase or abandon their cart? Did they activate the new feature or ignore it? The answer is always yes or no, true or false, one or zero.
When you aggregate binary metrics across many users, you get proportions and rates. The share of users who streamed becomes your weekly active user rate. The share of free users who upgraded becomes your conversion rate. These proportions are powerful because they're easy to interpret and directly tied to user behavior milestones.
**Example**
**Binary metrics in practice:**
For Spotify, you might measure whether each user streamed at least once this week (creating a weekly active user metric), or whether they have a premium subscription (creating a premium subscriber rate).
For e-commerce, the classic binary metric is conversion: did the user complete a purchase during this session? Averaged across all sessions, this becomes your conversion rate.
For a SaaS product, you might track whether each customer activated a specific feature within their first week, or whether they're currently on a paid plan versus a free tier.
## Ratio metrics
Beyond continuous and binary metrics, there's a third important type: **ratio metrics**. These metrics express a relationship between two quantities: clicks per impression, streams per session, revenue per order, or conversion rate per visit.
Ratio metrics are powerful because they normalize for opportunity. "Total clicks" might increase simply because you showed more content, but "clicks per impression" reveals whether users actually engaged more with what they saw. This normalization makes ratios particularly useful for comparing experiences where exposure varies.
**Example**
**Ratio metrics in practice:**
For Spotify, you might measure "streams per session" rather than just "total streams." This accounts for differences in how often users open the app and focuses on engagement depth within each visit.
For an e-commerce site, "add-to-cart rate" (carts per product view) is more informative than "total add-to-carts" because it controls for how much browsing happened.
For advertising, "click-through rate" (clicks per impression) is the standard metric because it normalizes for how many ads were shown.
### Analysis unit versus randomization unit
Ratio metrics introduce an important complexity: the **analysis unit** (what appears in the numerator or denominator) may differ from the **randomization unit** (how you assigned users to treatment groups).
Consider an experiment randomized at the user level measuring "clicks per impression." You assigned users to treatment groups, but the metric is computed per impression. Each user contributes multiple impressions, creating a mismatch between randomization and analysis units.
This matters because the core problem is that impressions from the same user are correlated—they share that user's preferences, tendencies, and session context. Standard statistical methods assume observations are independent. When impressions are naively treated as independent observations, this inflates your effective sample size, producing overconfident confidence intervals and falsely small p-values. The apparent weighting imbalance between high-volume and low-volume users is a symptom of this; the underlying cause is the violated independence assumption.
**Note:** When the analysis unit differs from the randomization unit, you need specialized statistical methods. Understanding the distinction helps you interpret results correctly and avoid common pitfalls in metric design.
**In Confidence:** Confidence handles the unit mismatch automatically, applying the delta method when your metric's analysis unit differs from the randomization unit.
**Example**
**The unit mismatch problem:**
You're testing a new ad format, randomized by user. Your metric is "click-through rate" (clicks per impression).
User A sees 100 impressions and clicks 5 times (5% CTR).
User B sees 10 impressions and clicks 1 time (10% CTR).
Naively treating each impression as an independent observation ignores that User A's 100 impressions are all correlated with each other — they're from the same person. The result is an inflated effective sample size, which makes the experiment look more powered than it is and produces overconfident confidence intervals. The right approach is to analyze at the randomization unit (user), not the analysis unit (impression).
For experimentation, you typically want user-level averages because that's your randomization unit. Understanding this distinction helps you interpret what your ratio metric actually measures.
**Quiz: What distinguishes events and measurements from metrics?**
- Events and measurements are qualitative, while metrics are quantitative
- Events and measurements are predictions, while metrics are historical facts
- Events and measurements are individual data points, while metrics are aggregations across users or sessions (correct)
**Quiz: Which of the following is a binary metric?**
- Share of users who created a playlist (correct)
- Average session length in minutes
- Total streams per user per week
**Quiz: Why do ratio metrics require special statistical treatment compared to continuous metrics?**
- Ratio metrics are harder to interpret
- Ratio metrics have higher variance by definition
- The numerator and denominator each carry independent variance, requiring methods like the delta method (correct)
## Notes for nerds
**The "continuous" label is a simplification.** Strictly speaking, metrics like streams per user are discrete counts, not truly continuous values. Count metrics and genuinely continuous measurements are grouped together here because from a statistical perspective—for mean-difference estimators—they require the same treatment. The meaningful distinction for analysis purposes is whether the outcome is numeric (count or continuous) or binary.
**Metric types and statistical methods.** Continuous, binary, and ratio are often presented as three distinct metric types in the experimentation literature, but they are not mutually exclusive categories. A metric can be both a ratio and produce a binary-style proportion (click-through rate, for instance, is clicks ÷ impressions and yields a value between 0 and 1). The reason these three are treated separately in practice is that they require different statistical methods: continuous/count metrics use standard mean-difference estimators; binary metrics use proportion-difference estimators; and ratio metrics require the delta method because their numerator and denominator each carry independent variance.
**The delta method and unit mismatch.** The unit mismatch problem described in this lesson—where the analysis unit (impression) differs from the randomization unit (user)—requires the **delta method** to compute variance correctly. Treating each impression as an independent observation ignores the correlation between impressions from the same user, which inflates your sample size estimates and produces overconfident results.
The delta method approximates the variance of a ratio metric X/Y using a first-order Taylor expansion around the means μ_X and μ_Y. This is the standard approach in large-scale experimentation platforms.
For a thorough treatment of the delta method applied to online experimentation metrics, see Deng, A., Lu, J., & Wang, S. (2018). "Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas." *Proceedings of KDD 2018*.
**Ratio metrics and variance reduction.** Ratio metrics don't just complicate variance estimation—they also make variance reduction more involved. When you apply regression adjustment (like CUPED) to a ratio metric, you can't simply adjust the ratio directly; you need to account for the joint behavior of the numerator and denominator. [Lesson 8: Variance reduction](./variance-reduction) covers this in depth.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-metrics/course-wrap-up
# Course wrap up
**Congratulations! You have finished Intro to metrics!**
You now have a comprehensive framework for designing and selecting metrics that drive better product decisions. You learned how to define metrics that capture the right user behavior, choose appropriate measurement approaches, and construct balanced metric suites for experiments.
As you apply these concepts in your work, remember that great metrics are:
- Aligned with clear hypotheses
- Interpretable by stakeholders
- Sensitive enough to detect meaningful changes
- Balanced between speed and strategic value
Go back to [my learning page](..) to continue your experimentation journey!
You should explore the [Intro to experimentation course](../../bootcamp/intro-course/introduction) next to see how to apply these metric design principles in practice.
Thank you for taking this course and helping to build a stronger experimentation culture!
---
## Scientific product development
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/introduction
# Scientific product development with experimentation
[Interactive: CourseProgressBar]
## Welcome
This course covers an introduction to evidence-based product development—using
the scientific method to learn about our end users and make product decisions.
The following pages cover the scientific method behind experimentation, how this
is applied at Spotify, and how you can apply the scientific thinking to your
product.
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material.
## Lessons
This course consists of 9 lessons of which one is a case study:
- Lesson 1: Why you should experiment: Learn about the benefits of experimentation and how it can help you make better decisions.
- Lesson 2: The scientific method: Learn about the scientific method and why it is useful in product development.
- Lesson 3: Randomized controlled trials: Learn about randomization of the treatment assignment and the role it plays in experimentation.
- Lesson 4: Experiment hypothesis: Learn how to specify a precise and testable hypothesis for your experiment.
- Lesson 5: Case study — shuffle button in Spotify: Practice hypothesis creation on a real Spotify experiment.
- Lesson 6: Why do we need statistics: Learn how statistics helps you quantify the uncertainty and make risk-informed decisions.
- Lesson 7: Success metrics: Learn how to select metrics and how to configure the sensitivity to detect effects.
- Lesson 8: Detectable effects and sample size: Learn how to set the sensitivity of your experiment using the minimum detectable effect and how that affects the sample size requirements.
- Lesson 9: Make a decision: Learn how to make decisions in a scientifically sound way.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/decision-making
# Lesson 9: Make a Decision
> In this lesson, you learn about decision making in the context of experimentation.
To benefit from experimentation in your decision making you should:
- Have a pre-determined decision rule
- Ship successful variants, iterate on non-successful variants using explorations and experiments
## Define a decision rule before you run the experiment
It is important to have pre-defined decision rule that maps any possible outcome
of the experiment to a product decision. For example, what will you do if one
guardrail metric moved in the wrong direction, but everything else looks good?
Predetermining the rule makes it easier to not change the goal after you see the results.
**In Confidence:** In Confidence, there is a default decision rule that gives decision recommendations throughout the experiments. Read more about how the recommendations are constructed in [the documentation](./../../docs/analyze-an-experiment).
## Iterate on a product with experimentation
The following chart shows how to apply the scientific method when you make a decision based on an
experiment.
If the experiment confirms the hypothesis and there is evidence that your change works well, then
you can proceed and roll out the change to all users.
If the experiment is not successful, that is, if there is a negative effect
detected on a guardrail metric or no success metric has improved, you should not
roll out the change. Instead, you should try and
understand why this iteration didn't have the intended effect, fix the problem,
and then re-run the experiment to see if your new fix actually fixed the
problem.
There are several ways of trying to understand why an iteration didn't have the
intended effect. For example, you can do exploratory analysis by diving into segments and additional metrics to better
understand your results. It is also common to do user research to get more in-depth understanding of how users experienced the change. All this information can then be used to formulate a
new hypothesis, and iterate on the product. You then test the new iteration
in a new experiment.
**In Confidence:** In Confidence, you can do exploratory analysis directly in the explore tab, diving into segments and additional metrics to better understand your results.
Although exploratory research and analysis is an important and natural step to inform new iterations,
don't use it to make a decision on the finished experiment. Changing the prediction to match the results invalidates the conclusion.
Of course, it is not a good idea to iterate forever. If repeated experiments fail to show improvement, that is a signal to abandon the hypothesis rather than keep refining it.
**Quiz: What should you do if you detect a negative effect on a guardrail metric?**
- Don't rollout and start a new iteration by trying to understand why it didn't have the desired effect. (correct)
- Ship the feature with a rollout. You have to break some eggs to make an omelette.
- Run the experiment again to see if the results are consistent.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/detectable-effects-and-sample-size
# Lesson 8: Set the sensitivity of the experiment with the minimum detectable effect (MDE)
> In this lesson, you learn about the minimum detectable effect (MDE) and how you use it to set
the sensitivity of an experiment.
The Minimum Detectable Effect (MDE):
- Decides how small effects in your metrics you can detect.
- The smaller MDE you choose, the larger the sample size you need for your experiment.
After deciding which metric to use to measure success, you need to define what effect size you want
the experiment to be able to detect. This effect size is called the "minimum detectable effect"
(MDE), or "minimum relevant effect". Use the MDE to set up the experiment so that it has enough
sensitivity to detect meaningful effects.
Picking the MDE is a trade-off between:
- the smallest effect relevant for the business
- the smallest effect that's practically measurable with the sample size you can reach in your experiment
As an experimenter, you can use your domain expertise and discuss with stakeholders to decide what
is the smallest effect that you would consider meaningful. In the next step, you calculate what
sample size you need to be able to reliably measure this effect. If the sample size required to
measure the chosen MDE is unrealistically large, then you need to adjust the MDE upwards.
One way to understand the minimum detectable effect (MDE) of an experiment is to imagine your
experiment as a microscope.
## Illustration: MDE as the resolution of a microscope
Imagine looking at tissue-sample under a microscope. The more you zoom in, the
more details you can see. Changes to the sample that would be hard or impossible to see at one
level of magnification become clear at a higher level of magnification. The
minimum detectable effect (MDE) in an experiment is like the resolution of a microscope. It is
the smallest change that you want to be able to see. If you want to be able to
see smaller changes, you need a higher resolution. In experiments, you can
increase the 'resolution' or sensitivity by increasing the sample size. The larger the sample,
the smaller changes you can detect. Just like you cannot zoom in on a microscope
indefinitely, you cannot detect arbitrarily small changes with an experiment,
because you don't have an infinite number of users.
**Learn about the Minimum Detectable Effect (MDE) and how to set it in your experiment in 3 minutes and 43 seconds.**
[Video: https://www.youtube.com/watch?v=lV50V6hj2ms]
**Quiz: What is the purpose of the minimum detectable effect (MDE) in an experiment?**
- Determine the smallest effect that is relevant for the business and use it to inform about how many users the experiment needs to reach. (correct)
- To set the right settings for microscope in any laboratory.
- To minimize the detectable effect of the experiment to avoid mistakes.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/experiment-hypothesis
# Lesson 4: Experiment hypothesis
> In this lesson, you learn how to create a plan for your experiment.
You learn how to formulate a hypothesis that acts as the product foundation for your experiment,
and get examples of essential questions to ask yourself when planning your experiment so that you
don't run into problems later on.
A good hypothesis:
- Is short and direct.
- Is testable in the sense that the experiment can give clear evidence for or against it.
## Define your hypothesis
Before you run an experiment, you need to formulate a _hypothesis_ statement. Use it to articulate
what you plan to test, and how. When you run an experiment, you actually do hypothesis testing, so
this step is important!
The process of formulating a hypothesis allows (or forces) you to think through the basis for what
you are testing, and put this into writing. A well formulated hypothesis should contain:
- What prior information led to this hypothesis.
- What change you make.
- For whom (typically which users) you make the change.
- What you hope the change achieves.
- How you plan to decide whether it was successful.
### Hypothesis template
A template that you can use to formulate this is:
**Based on** [prior knowledge], **we believe that** [theory about user need]**. We think that**
[doing this/building this feature/creating this experience] **for** [these people/personas] **will
achieve** [these outcomes]**. We will know this is true when we see** [metric results].
**Note:** Using a change in the sign-up flow as an example, you could formulate a hypothesis as follows:
**Based on** *user research,* **we believe that** *having to create a username creates friction in the signup
process.* **We think that** *removing the step to enter a username* **for** *users signing up in
the app* **will lead to** *more users successfully completing the signup flow.* **We will know
this is true when we see** *an increase in the sign-up completion rate*.
A strong hypothesis should also describe _why_ you believe this change will achieve the desired
outcome. You should back it up by earlier research, data, or domain knowledge (and not just
base it on a hunch).
\*Go from Goal to Hypothesis, adapted from the Thoughtful Execution framework.
**Note:** In a later lesson, you will learn more on how to define success, including how
to select success metrics, and how to think about **at what point you will
consider a change in a metric to be a sign of success**. For example, by how
much does the sign-up completion rate need to increase for you to consider it
a success—by 1%? Or as little as 0.1%? More about this later!
Ideas, that could become fully defined hypotheses, can come from anywhere—an engineer, a designer,
customer support, or an end-user of your product. Many ideas could result in product changes and new
features. Without testing them, you won't actually know if you were correct and that the change
in fact made the product better. Experiments help you do that!
## Make sure you're good to go
When the hypothesis is starting to take shape, it's time to also consider things like:
- How do you plan to build the experience that you want to test? Who do you need to involve to make it happen?
- Do you need to sync with any other teams about what you are doing? For example, are you using,
modifying or impacting part of your product that another team owns?
- Do you need to coordinate your experiment with any current or future other activities?
Doing this kind of thinking and planning early on can save you a lot of time and effort later on!
**In Confidence:** Are the metrics you want to track already available in Confidence, or do you need to [set them up](../../docs/metrics/quickstart)?
**Quiz: Which of the following do you consider to be the most complete and testable experiment hypothesis?**
- We believe that a simpler signup flow will lead to more users signing up to Spotify.
- We believe that a simpler signup flow will lead to more users signing up to Spotify. We will know this is true when we see an increase in the sign-up completion rate.
- We believe that removing the step to enter a username for users signing up in the iOS app will lead to more users successfully completing the signup flow. We will know this is true when we see an increase in the sign-up completion rate. (correct)
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/rct
# Lesson 3: Why you need randomized controlled experiments
> In this lesson, you learn about the role of randomization in experiments. Randomized experiments are
also known as randomized controlled trials.
Randomized treatment assignment:
- Makes treatment groups similar in all aspects besides which treatment they receive.
- Makes it possible to interpret the treatment effect as the _causal_ effect of the treatment on the outcome.
## Example: The effectiveness of a weight-loss program
Let's imagine that you own a gym, and you want to offer a weight-loss program to your customers. You
want to know how effective your program is, so you design an experiment.
### Experiment design 1: Uncontrolled trial
You stand at the entrance of your gym and look for volunteers to participate in your program so you
can measure its effectiveness. You weight the people that want to join your program before they
enroll. After a 6-week program, consisting of exercise schedules and dietary advice, you weigh them
again. You calculate the average difference in weight before and after the program, and find that
people lost 3 kg on average. You celebrate a great success and start advertising your program!
#### The problem with an uncontrolled trial
You don't know what would have happened if people didn't enroll in your program. All participants
wanted to lose weight, and maybe they would have done so without your program. People's
weight fluctuates over time. People who had just gained some weight (for example after holidays) may
be more motivated to sign-up. They may also just lose weight again just because they returned to
their normal lifestyle. If you want to know the effectiveness of your program, then you need to
compare your program with a situation without it.
### Experiment design 2: You need a control group
You stand at the entrance of your gym and look for volunteers to participate in your program so you
can measure its effectiveness. You enroll the people that want to join. You ask the people that
don't want to participate to be part of a control group. You weigh both groups before and after the
program. After the program, you calculate the change in weight before and after for each group,
and then calculate the difference between both groups. You find that our treatment group lost more
weight than the control group! You celebrate a great success and start advertising our program!
#### The problem with observational control groups
Our treatment and control groups are not comparable. The treatment group wanted to lose weight, and
the control group didn't. The control group may contain people who joined the gym to become
stronger, and may have even gained weight from growing muscles! This mechanism is called
[selection bias](https://en.wikipedia.org/wiki/Selection_bias), and happens when groups are selected in a way
that biases the result. Selection bias causes incomparable groups and invalidates any result. To
avoid selection bias, you need a method of assigning people to the treatment and control groups,
that can't have any correlation with the outcome that you plan to measure.
### Experiment design 3: Randomized controlled trial
You want to get a precise estimate of the effectiveness of your weight loss program. For this, you
need to compare people who took the program to a control group that is comparable in all other
relevant aspects, except for the fact that they have taken the program. For this example, you could
do the following instead. You again stand at the entrance of your gym and ask people if they are
interested in participating in your weight-loss program. If they say "no", then they don't
participate in the trial. If they say "yes", you weigh them and then flip a coin. Based on the coin
flip you either enroll them right away, or you tell them "The program starts in 6 weeks, come back
then!"". This way you remove any selection bias. The random assignment makes sure that on average,
the groups are similar across all other characteristics except for the treatment that you give them.
Of course, you still need to make sure that you can collect the data from everybody in the treatment
and the control group after 6 weeks!
## Randomized controlled trials
Experiments, like A/B tests and rollouts, split users into two (or more) groups by
random assignment. The random assignment makes sure that the groups are, on average, similar in all
aspects except for the change you want to test. For example, if you randomly split all Spotify users into two
groups, the two groups should be very similar in terms of dimensions like demographics, connection
speed, and music taste. One group gets the status of a "treatment" group and receives the new
feature. The other group receives the default feature. you can then observe the users over time while
they receive two different experiences, and measure some outcome of interest, for example churn,
daily activity, or the number of minutes played. At the end of the experiment, you run a statistical
test to calculate whether the differences between the groups are larger than what you expect to see
if there's no difference.
**Quiz: What is the purpose of randomizing the treatment assignment in a controlled experiment?**
- To ensure that the results are significant.
- To limit the time effects in the analysis.
- To make the groups comparable in all other aspects than which treatment they received. (correct)
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/shuffle-example-1
# Lesson 5: Case study: Shuffle button in a shelf on Spotify Home
## Write a hypothesis for an experiment that adds a shuffle button to a shelf on Spotify Home
This case study is based on an actual experiment at Spotify. The details have
been modified for the purpose of this exercise. The experiment adds a
shuffle button to the 'Try something else' shelf on the home screen of Spotify.
The text below gives some information on earlier research, theory, and motivation that led to
an experiment which tested the impact of adding a shuffle button on the 'Try something else' shelf on the home screen.
Use the information below to formulate a testable hypothesis for this experiment.
### Prior knowledge
- In earlier user research, we found that **Just play something** actions decreased the perceived friction to start listening.
- A large part of the consumption from Spotify is via surfaces that automatically select what audio to play.
- [Anecdotal evidence](https://this.deakin.edu.au/self-improvement/link-between-stress-watching-tv) that the stress of having to choose can explain why users continue to consume the same content.
### Theory about user need
Providing the user with low-effort paths in terms of making a decision can
decrease the load on the user to start listening, and thereby increase engagement and audio
consumption.
### Motivation
- If users get stuck in browsing for content, and are unable to make a decision, they might leave the app.
- Offering a low-effort path to start listening will decrease perceived stress and can increase engagement and audio consumption from the home screen.
- Similar features have been successful in other places in the app before.
## Exercise: Write a testable hypothesis for this experiment
1. Which success metrics would you set for this experiment? Why would you choose these metrics?
2. What unintended side effects could you see when you add a shuffle button to this shelf? What
guardrail metrics would you set to test for this?
3. Based on the information above, write a testable hypothesis for an experiment on
a shuffle button on the **Try something else** shelf. You can use the template below as a starting point.
You can find answers to these questions on the next page.
### Hypothesis template
Based on **_[prior knowledge]_**,
we believe that **_[theory about user need]_**.
We think that **_[doing this/building this feature/creating this experience]_**
for **_[these people/personas]_**
will achieve **_[these outcomes]_**.
We will know this is true when we see **_[metric results]_**.
This will be good for **_[customers/artists/ our business]_** because **_[motivation]_**.
## Bonus questions
4. How much of an increase in these metrics would you want to see to call the experiment a success?
5. What is the largest decrease in the guardrail metrics that you would still consider acceptable?
You may not be able to give an exact number for questions 4) and 5), but think about how you
would come up with a number to answer these questions.
**Quiz: What is the primary reason for selecting 'Minutes played on week 1 after exposure' as a success metric instead of 'Consumption from the shelf with the shuffle button'?**
- To avoid inflated results from novelty effects, which are more pronounced for shelf-level metrics than for overall consumption.
- To specifically measure the impact on overall audio consumption, aligning with the goal of increasing engagement. (correct)
- Because shelf-level consumption metrics have higher variance than overall consumption metrics, requiring a larger sample size to reach significance.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/shuffle-example-1-answers
# Answers to Case study: Shuffle button in a shelf on Spotify home
## Exercise: Write a hypothesis for an experiment
1. Which success metrics would you set for this experiment? Why would you choose these metrics?
The product brief states the theory that the shuffle button will _"increase engagement and audio consumption"_. We probably want to measure a metric that measures consumption, such as _Minutes
played on day 1 after exposure_. However, there could be a strong novelty effect that leads to
increased consumption on the first day when the button is new to the user, but wears off quickly.
Therefore _Minutes played on week 1 after exposure_ may be a better choice to measure this effect.
To be sure that the effect is persistent over time, we could add more metrics for other time points,
such as _Minutes played on week 2 after exposure_. However, adding more metrics can make it more
difficult to detect a change. Thinking carefully and then picking fewer metrics is often the better
option.
2. What unintended side effects could you see when you add the shuffle button to this shelf? What
guardrail metrics would you set to test for this?
There is a risk that adding the shuffle button will increase crashes. The
metric _Share of users with a crash on day 1 after exposure_ would be a good
metric to detect if this happens. In addition it might be that we are only
moving consumption from other shelves to the shelf with the shuffle button.
However, we don't need to add a metric for this, since we are measuring overall
consumption. If consumption is only moved around, we will see no effect in the
experiment. If instead we used _Consumption from the shelf with the shuffle
button_ as a metric, we would need to add a metric for _Consumption from other
shelves_ to make sure that we are not just moving consumption around.
3. Based on the information above, write a testable hypothesis for an experiment on
a shuffle button on the **Try something else** shelf.
Based on user research, consumption data and anecdotal evidence we believe that
giving users low-effort paths to discovering new content is important for user
satisfaction and engagement. We think that adding a shuffle button to the 'Try
something else' shelf for users in Spanish-speaking Latin America will achieve
increased overall audio consumption. We will know this is true when we see an
increase in _Minutes played on week 1 after exposure_. This will be good for
customers because they can discover new songs and artists without having to make
decisions and for creators because more users will discover more content.
## Bonus questions
4. How much of an increase in these metrics would you want to see to call the experiment a success?
When setting up an experiment, we need to define the "minimum detectable effect"
for each success metric. This helps you set up the experiment to be sufficiently
sensitive to measure the effect that we care about. It is not always
straightforward to define how large an effect needs to be relevant. In this
case, there is historical data on similar experiments in other places, from
which we can learn what size of lift to expect.
5. What is the largest decrease in the guardrail metrics that you would still consider acceptable?
For a metric like _Share of users with crashes on day 1 after exposure_, even a
small increase may be unacceptable. If the change increases the number of
crashes, we would probably want to fix the source of the crashes, rather than
rolling the change out despite causing crashes to some users. So we may want to
set the minimum detectable effect very low here.
On the other hand, if we set the effect very low, then we will need a large sample size to
detect an effect this small.
At the same time, the baseline
of crash rates is low, so even a large relative increase in crash rates may not
be a large absolute increase.
Setting a sensible effect size here is a trade-off between an effect that is
* small enough to protect the business against rolling out a change that can cause a
negative experience
and
* large enough to be able to detect it within a reasonable sample size.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/success-metrics
# Lesson 7: Success metrics
> In this lesson, you learn how to define success metrics for an experiment. Success metrics are what
you use to evaluate whether your change is successful.
A good success metric:
- Has a strong relation (ideally direct) with success of your product and business.
- Is not too noisy and therefore easy to detect changes in.
After you've written the hypothesis, you should have a clear idea which user behavior the
experiment should influence and what outcome you expect to see. Now you need to pick metrics
that measure if the experiment successfully achieves this outcome. An ideal success metric
directly measures the desired outcome and is
- Observable in the short term
- Sensitive to changes
- Relevant for the business in the long term
In the best case, you can measure your desired outcome directly and with a reasonable delay after a
user's exposure to the change.
**Example**
Consider an example that makes a change in the user flow for subscribing to
premium. The experimenters can measure the share of users who successfully
sign up. The impact on user behavior is directly related to the change in the
user flow, it's measurable in the short term, and highly relevant to the
business.
Unfortunately, often the outcome of interest happens further in the future and is difficult
to measure directly in the experiment.
**Example**
For example, when we create a new feature at Spotify, we often hope to improve
the user experience and reduce churn in the long term. But the subjective user experience is difficult to
measure, and the impact of the user experience on churn takes time to detect. In those
cases, we need to use proxy metrics that we can measure in the short term, and are reliable predictors of
the long-term outcome that's our primary interest.
**Example**
At Spotify, common proxy metrics are *share of active users* (measured over a
day or a week) and *minutes played*. These metrics measure short-term
engagement with the product and correlate with long-term outcomes like churn
and premium subscription.
## Select few specific metrics
Success metrics should be as specific to the hypothesis as possible.
You may be curious to learn about all the possible effects that your treatment may have. It's often
tempting to just add every single metric that your change could possibly impact. However,
when deciding on a success metric you should limit yourself to a few relevant metrics,
and separate explorations from the criterion that defines success.
You should select only few success metrics because:
- It's harder to reliably measure success with many metrics
- More metrics require a larger sample size
After your experiment ends, you can explore the effects on other metrics using exploratory analysis. This can
help you understand the results better and inspire new hypotheses. However, you should base the decision whether
to ship a change on your pre-defined success metrics, not on metrics that you
added afterwards. Pre-defining decision criteria helps to avoid [confirmation bias](https://en.wikipedia.org/wiki/Confirmation_bias), where you end up selectively
looking for evidence that confirms your beliefs and ignore evidence against.
**In Confidence:** In Confidence, you can run [exploratory analysis](../../docs/exploration) after your experiment ends to dig deeper into results and get inspiration for new hypotheses.
### Example
**Example**
Consider a team that's working on the Spotify home page that wants to test whether adding a "shuffle" button in
the "Try something else" shelf increases user engagement on the home screen. They create an
experiment with two treatment groups: one called "Control" which gets the default experience (no
shuffle button), and one called "Treatment" which gets the shuffle button.
They need to decide on a success metric to decide whether the shuffle button improves user
experience. Which metric should they choose?
If the goal of the button is to increase interaction with the **Try something else** shelf, then one
possible metric is _Share of users who play from the **Try something else** shelf_. This
directly measures the behavior that the feature aims to influence. But is this also relevant for the
user and the business? Measuring success by that metric makes it tempting to introduce more features
that direct traffic towards this shelf, and away from the **Jump back in** and **Podcasts to try** shelves. A better
success metric is _Minutes played on Week 1_, because this measures overall user activity.
You could add _Share of users who play from the **Try something else** shelf_ as a metric to confirm
that an increase in plays from **Try something else** caused an increase in overall activity.
**Quiz: What is the purpose of a success metric in an experiment?**
- To validate statistical assumptions
- Evaluate if the product change had the intended impact (correct)
- Check that the experiment is correctly configured
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/the-scientific-method
# Lesson 2: The origin of the scientific method
> In this lesson, you learn about the scientific method and what makes evidence
from this method different from other types of evidence, and why you should
regard it as the most trustworthy way of learning about the world.
The scientific method is old. One classic example of the essence of the scientific method and how it
differs from other ways of learning about the world is Halley's work on comets.
## Example: Halley's comet
Once every 76 years, the comet that is now known as "Halley's comet" passes close to the Earth and
can be seen with the naked eye. Halley's comet has been observed since at least the year 240 BC, but
it is only since 1704 that we know that this is the same comet that returns again and again. Since
Aristotle, scholars believed that comets were disturbances in the Earth's atmosphere. In 1577 the
Danish astronomer Tycho Brahe found that comets passed behind the moon, and could therefore not be
part of the atmosphere.
In 1704, Edmund Halley used the methods published by Newton to calculate the orbits of comets
that had been observed in the centuries before. He found that the comets that had been seen in
1531 and 1607 had the same orbit as a comet that had been seen in 1682. Based on this
observation, Halley formulated a hypothesis:
**_"I observed that the comets from 1531, 1607, and 1682 have the same orbit. Based on this
observation, I believe that this is in fact the same comet, that moves in ellipses around the
sun and returns once every 76 years. I predict that this comet will be seen again in 1758."_**
Or, in his own words: _"Hence I dare venture to foretell, that it will return again in the year
1758."_
Halley died before he could see his prediction verified. The comet returned on schedule, in 1758
to the amazement of the scientific community, the public and the British _The Gentlemen's
magazine_, which wrote:
_"By its appearance at this time, the truth of the Newtonian Theory of the Solar System is
demonstrated to the conviction of the whole world, and the credit of the astronomers is fully
established and raised far above all the wit and sneers of ignorant men."_
**Question:**
Would Halley's prediction have been as impressive if he had predicted:
"I predict that this comet will be seen again some time between 1740 and 1770?"
Would Halley's prediction have been as impressive if we saw comets almost every year?
**The observation of the comet at the predicted time provided fairly strong evidence for his
hypothesis, because it was very specific.** Predicting the next observation of a comet and
getting it right by pure luck would have been very unlikely.
## The scientific method
The scientific method has five basic steps, plus one feedback step:
1. Make an observation
2. Ask a question
3. Form a hypothesis, or testable explanation
4. Make a prediction based on the hypothesis
5. Test the prediction
6. Iterate: use the results to make new hypotheses or predictions
## What is an experiment?
An experiment is a procedure designed to test a hypothesis as part of the **scientific method**.
There are different types of experiments:
### Uncontrolled experiments
An uncontrolled experiment involves making a prediction or forming a hypothesis and then gathering
data by observing a system. The variables are not controlled in a natural experiment. A historical
example is Edmund's Halley's hypothesis about the orbits of comets described above.
### Controlled experiments
In a controlled experiment, you compare a treatment group with a control group, to test the effect
of a treatment on an outcome. Ideally, the two groups are the same except for the change in treatment.
A historical example is the
[Salk Polio vaccine trial](https://www.history.com/this-day-in-history/salk-announces-polio-vaccine),
which treated 600,000 children with either the new vaccine or placebo. A/B tests and rollouts are controlled experiments.
**Quiz: What is the purpose of the scientific method?**
- To ensure accurate and reliable results. (correct)
- To communicate findings to other scientists and the public.
- To make experiments more interesting and engaging.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/why-do-we-need-statistics
# Lesson 6: Why do we need statistics?
> In this lesson, you learn about the role that statistical analysis plays in experimentation.
Statistics helps to:
- Quantify the uncertainty in the metric results.
- Makes it possible to manage and bound the risk of making the wrong product decisions.
Statistics is the mathematical language for quantifying uncertainty. In experiments, there is always
random variation between the treatment groups even before you make a change to any group of users.
All users are unique and therefore treatment groups never have _exactly_ the same average.
## Gym example
Let's go back to the weight-loss program trial.
You carried out your experiment as planned, randomizing participants into a control group and a
treatment group. You measured the weight of the participants in both groups at the start of the
trial. The treatment group started the 6-week weight-loss program right away, while the control
group waited 6 weeks. At the end of the 6 weeks, you measured the weight of everyone again.
You ended up with 10 participants in each group, and the table below shows the difference before and
after 6 weeks. People in the control group lost 1.3 kg on average, and people who participated in
the weight-loss program lost 3.1 kg on average.
| Weight change (kg) | **Group average** | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| ------------------ | ----------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Control | **-1.3** | +2.1 | -3.3 | -2.7 | -1.8 | +2.1 | -1.8 | +0.9 | -1.2 | -4.5 | -3.0 |
| Treatment | **-3.1** | -3.9 | -2.7 | -3.3 | -7.5 | -0.9 | -4.2 | -1.8 | -2.1 | -3.9 | -0.6 |
So, the people in your program lost 1.8 kg more weight than the people in the control group.
Does that mean that your program works? Or could it just be a coincidence?
## Quantify the noise to detect the signal
People's weights fluctuate somewhat over time. If you just divided 20 people randomly into two
groups and measured their change in weight over time, it's quite unlikely that the averages of the
two groups would be exactly the same. This random variation causes noise in your measurement.
**How can you detect the signal among the noise?**
To answer this question, you can think about how much "noise" in the weight measurements you would
expect to see, even if your program had no impact at all. You can then compare the difference that you
found, to the amount of noise that you would expect purely from random variation.
The amount of noise in the average measurements depends on the number of people in each group, and
the _variance_, that is how much the individuals differ between each other. Based on this, you can use statistical
theory to calculate how much _noise_ you can expect in the measurement. In other words: **How likely
would it be to find a difference this large, just due to random variation**.
We'll
[skip the math](https://en.wikipedia.org/wiki/Welch%27s_t-test) here.
For the weight-loss example, it turns out that there is an 8% chance to find a difference between two
groups of 1.8kg or more, based purely on random variation. This calculation assumes that your
weight-loss program had absolutely no effect on people's weight.
## Use statistics to make a decision
You can use this calculation to make a decision about your program.
If you find quite a small effect it may not be unlikely to see it even if your
program did nothing. We can conclude that there's not enough evidence to conclude that your
program is working.
If you find a larger effect, it is less likely to see this purely due to random variation.
We can define a threshold beforehand, to decide how _"unlikely to be seen purely based on random
variation"_ your result needs to be, to consider it **strong enough evidence**.
For example, if you had decided beforehand that you would consider your results strong enough if
they are less than 5% likely to be seen based purely on random variation, then the result of your
trial would not have produced strong enough evidence. If on the other hand, you had decided
beforehand to set that threshold at 10%, then your result would have passed the test. We call a
result that has passed such a test "statistically significant".
### Don't change the target after you see the results
It is tempting to change the threshold after seeing the results. "OK, we said 5% beforehand, but 8%
is still not that bad, if we decide that a threshold of 10% is good enough, then the experiment
passes the test!". But to avoid confirmation bias, it is important to define the threshold before
seeing the results. Changing the threshold after seeing the results, is like shooting arrows at a
wall, and then drawing a target around the place where the arrow landed. Drawing a target around the
results of an experiment is cheating, and cheating in experimentation leads to worse product
decisions. The same applies to changing the metrics after the end of the experiment.
## A note on statistical uncertainty for the curious
Statistics doesn't magically know how different any two groups are. However, you have one trick up
your sleeve: **randomization**. By randomizing the treatment assignment, you know how the difference
in means between two treatment groups varies across different random treatment assignments. In other
words, randomly assigning the treatment to users serves two purposes, make the groups similar in all
other aspects than the treatment (as discussed in the scientific method lesson), and to 'structure'
the noise in the difference-in-means estimator to allow statistical inference.
**Quiz: What is the purpose of statistical testing?**
- To ensure that the results are successful.
- Quantifying uncertainty and limit the risks of reaching the wrong conclusion. (correct)
- Use p-values as much as possible to gain efficiency.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/why-you-should-experiment
# Lesson 1: Why you should experiment
> In this lesson, you learn when you should experiment. Run experiments to:
- Objectively test your own biased assumptions
- Avoid accidentally causing breakage while trying to improve
- Innovate fast by abandoning bad ideas early
- Establish a causal link between a product change and an outcome
## We experiment because we know that we have biases
As humans, we tend to look for evidence that supports what we already believe, a phenomenon known as
[confirmation bias](https://en.wikipedia.org/wiki/Confirmation_bias). To make matters worse, we also
have a tendency to overvalue the products that we built ourselves (also known as the
[IKEA effect](https://www.behavioraleconomics.com/resources/mini-encyclopedia-of-be/ikea-effect/)).
This means that if we want to know the true value of product changes for our users, we have to be
very careful to measure the impact in an unbiased and objective way, to avoid having our own beliefs
fool us.
## We experiment to avoid accidental breakage
Every time we change something about our product, we run the risk of accidentally causing negative
side effects. This could be an increase in latency or crash rates caused by a new feature. For
a mature product such as Spotify, it is much easier to unintentionally break the user experience than
to improve it. Without experimentation, small undetected decreases in performance can add up and
have a detrimental combined impact on the overall user experience. 
## We run experiments to innovate fast and abandon bad ideas early
The most important thing for most companies, Spotify included, is not to ship **a lot of changes**,
but to ship **the right changes**. To not release negative product changes is as important as to
release new positive changes to the product. Without testing our assumptions systematically on real
users in a real life setting, we risk investing a lot of development resources into product changes
that appeared promising at first, but didn't actually improve the user experience in a real life
setting.
## Experiments allow us to draw causal conclusions
Let's say that we are looking for ways to reduce churn for Spotify premium users. We could do an
analysis that compares users who churned with users who didn't. One result of such an analysis could
be that users who didn't churn experienced more app crashes than users who churned. Does this mean
that increasing the number of app crashes would reduce churn? Of course not. People who use the app
a lot are more likely to experience a crash, and are also less likely to churn.
Now let's imagine that we built a new feature, and we hope that it reduces churn for Spotify
premium users. In theory, we could just roll out the feature to everyone, check how many people are
using it, and then see if people who use the feature are less likely to churn. But would this tell
us if the feature actually reduces churn? No. Because just as with app crashes, a correlation
between more feature usage and less churn would not imply a causal link. To objectively measure the
value of our new feature, we need to find a way to isolate the impact of the feature from everything
else that can impact our metric of choice. The gold standard method for doing this is called a
"randomized controlled trial."
## Randomized controlled trials
Experiments split users into two (or more) groups by random assignment. The random
assignment makes sure that the groups are, on average, similar in all aspects except for the change
we want to test. If we randomly split all Spotify users into two groups, the two groups should be
very similar in terms of dimensions like demographics, connection speed, and music taste. One group gets the
status of a "treatment" group and receives the new feature. The other group receives the default
feature. We can then observe the users over time while they receive two different experiences, and
measure some outcome of interest, for example churn, daily activity, or the number of minutes
played. At the end of the experiment, we run a statistical test to calculate whether the
differences between the groups are larger than what we expect to see if there's no difference.
## The cost of experiments
Experiments aren't free. The main costs involved are that:
- It takes time to set up an experiment, wait for users to be exposed and analyze the results.
- If the change that you test is as beneficial as you hope, then the users in the control group miss
out on the improved experience until the end of the experiment.
- If a change makes the user experience worse, then some users receive a worse experience for as
long as the experiment runs.
## The cost of *not* running experiments
- You don't know if users respond to the product change in the way that you expect.
- You might have negatively impacted your users in unexpected ways. If you roll out many changes
without A/B testing them on real users, there might be negative impacts on system performance,
crash rates, and more that you fail to detect. Taken together, they can add up and seriously
impact the user experience.
- Without testing your assumptions on real users, you risk investing resources into
product changes that appear promising, but don't actually improve the experience in a real life
setting.
## Learn more
Watch this video to see
examples of different types of experiments for various common use cases.
[Video: https://www.youtube.com/watch?v=B6QFyLkWiP8]
**Quiz: Why should all changes that affect end-users be tested with A/B tests and/or rollouts?**
- To ensure that our changes have the effects we intended and detect unexpected side effects that might harm our end users and thereby our business. (correct)
- To inform other parts of the company what we are working on to ensure transparency.
- Experimentation is important in itself, because there could be no learning without it.
---
Source: https://confidence.spotify.com/bootcamp/scientific-product-development/course-wrap-up
# Course wrap up
**Congratulations! You have finished the Scientific product development course!**
Thank you so much for taking this course and helping to build an even stronger experimentation
community at your company!
Happy experimenting!
Go back to [my learning page](..) to keep learning!
---
## A primer on hypothesis testing
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/introduction
# Welcome to *A primer on hypothesis testing*
*A primer on hypothesis testing* is an asynchronous, self-paced course that
teaches you the fundamental concepts of hypothesis testing in the context of
experimentation. This course focuses on building intuition for how hypothesis
tests work and why they are crucial for making decisions in experiments.
In this course, you'll learn how hypothesis testing helps manage uncertainty in
experimental data, understand the relationship between what you observe in a
sample and what is true for the full population, and learn how to interpret
statistical significance and p-values.
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material. Complete each lesson's questions to track your
progress.
## Before you begin
Before you start this course, you should have a basic understanding of what it means to run an experiment end-to-end. In our experience, this helps you internalize the concepts in our courses.
**In Confidence:** If you haven't already, go through the [A/B test quickstart](../../docs/abtests/quickstart) to get hands-on experience running an experiment in Confidence.
## Lessons
This course consists of the following lessons:
- Lesson 1: What is hypothesis testing?: Learn the basics of hypothesis testing and how it helps handle uncertainty in experimental data.
- Lesson 2: True versus estimated difference in means: Understand the difference between population parameters and sample statistics in hypothesis testing.
- Lesson 3: The sampling distribution: Learn about sampling distributions and the Central Limit Theorem's role in hypothesis testing.
- Lesson 4: P-values and rejecting the null hypothesis: Learn about the interpretation of p-values and learn how to make decisions based on statistical significance.
- Lesson 5: False positive rate and alpha: Learn about the false positive rate and how it relates to the alpha parameter.
- Lesson 6: True positive rate and power: Learn about the true positive rate and how it relates to the power parameter.
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/fpr-and-alpha
# Lesson 5: False positive rate and alpha
> This page teaches you about the concept of false positive results and
the rate at which they appear in experiments. You learn:
- What a false positive result is
- What the false positive rate is
- What alpha is and how it relates to the false positive rate
## A false positive result
A false positive result, often simply called a "false positive," is when we find
a statistically significant effect from a treatment in an experiment when the
treatment actually doesn't have an effect. Another term for
a false positive result is "a type I error".
In most experiments, we are testing the mean difference between the treatment
groups. A false positive result in this case would be when we observe a large
enough mean difference between treatment and control groups to be statistically
significant even though the treatment had no effect on the outcome.
All users have some value on the outcome metric even if the treatment has no
effect. Some have large values relative to the population mean, some have small
values relative to the population mean. When we are randomly splitting the users
into treatment and control, there is always a risk that most users in the sample
with large values end up in the treatment group rather than in the control
group.
[Interactive: UserAssignmentSimulator]
---
## The decision from one experiment will be right or wrong
Although the random treatment assignment makes it possible to quantify the
variation, it also means that we can never be certain about whether an observed
result in an experiment is true or not.
[Image: Possible outcomes]
There is no way around the fact that we will never be certain about the result
of a single experiment. However, we can ensure that the rate of false positives
across many experiments is bounded at a certain level.
Statistical tests are constructed to limit the rate of finding the wrong result across many experiments.
---
## False positive rate
Valid statistical tests quantify the variability of the test statistic under the
hypothesis of no treatment effect and use that to bound the rate at which we get
false positive results. Only the alpha percent most unlikely imbalances under
the null will be considered significant.
A good property for a statistical test to have is that the rate of false
positives is bounded to a certain level which can be controlled by the
experimenter.
**Note:** The false positive rate is the rate at which we find false positives in experiments where there is no effect.
For example, if we run 100 experiments where the treatment has no effect on the
outcome metric, and we find that 10 have a significant effect, the proportion of
experiments where we find a significant effect (10/100 = 10%) is the false
positive rate of this test.
--------------------------------------------------------------------------------
## Alpha (the intended false positive rate)
Alpha is a parameter that statistical tests have that corresponds to **the
intended upper bound on the false positive rate**. We say that a statistical test
is valid if the false positive rate over repeated experiments (with no effect)
is lower than or equal to alpha. In other words, by using valid statistical
tests, we can bound the proportion of experiments where there is no true effect
but we falsely find one.
At this point, you might be wondering why we cannot simply set alpha to zero to
avoid all false positives. The reason is that we also want to be able to find
true effects when they are there, and unfortunately, there is a trade-off
between the false positive rate and our ability to find true effects which we
return to in the next lesson.
---
## False positive rate simulator
Now we can put what we have learned together and simulate experiments to see how
often we find false positives with a given alpha.
In this simulator, we are using a Z-test and are drawing large random samples.
Since the sample size is large, the distribution of the test statistic under the
null hypothesis is approximately normal and therefore the false positive rate
should be close to alpha across many random experiments.
[Interactive: FalsePositiveSimulator]
If we ran the simulation with infinitely many experiments, the rate converges on exactly alpha%.
---
**Quiz: What is a false positive result in an experiment?**
- When a statistically significant effect is found, but the treatment actually has no effect (correct)
- When the treatment has an effect, but the result is not statistically significant
- When the null hypothesis is true, but we fail to reject it
- When a statistically significant effect is found, and the treatment has an effect
**Quiz: What does the false positive rate represent?**
- The proportion of experiments where a significant effect is found when there is no true effect (correct)
- The percentage of experiments where the treatment effect is real, but not detected
- The probability of rejecting the null hypothesis when it is false
- The likelihood that an experiment produces the same result repeatedly
**Quiz: What is the role of alpha in hypothesis testing?**
- It controls the likelihood of correctly rejecting the null hypothesis
- It determines the intended upper bound on the false positive rate (correct)
- It represents the proportion of false negatives in an experiment
- It quantifies the variability of the test statistic under the null hypothesis
---
## Notes for Nerds
### Conservative tests
The test used in the simulation would reach exactly the intended false positive
rate if we simulated a large enough number of experiments. However, for a test
to be valid, it's enough that the false positive rate is **lower than or equal
to alpha**.
A statistical test that has a false positive rate substantially lower than alpha
is called **a conservative test**. Generally speaking, it is good to avoid
conservative tests as they give the experimenter less control over the risk
management of the experiments.
---
### Intended vs actual false positive rate
Note that we say "intended" false positive rate. If we use a statistical test
incorrectly, the actual false positive rate might not in fact be bounded by
alpha.
A classic example of when a test is misused causing inflated false positive
rates is when a fixed-sample hypothesis test is used to peek at the data
multiple times. In this case, the false positive rate is not bounded by the
alpha of the test, because the test only bounds the false positive below alpha
if the test is performed once at the end of the experiment, not if it is
performed repeatedly.
Read more about the issue with peeking on standard statistical hypothesis tests in the
[blog post on sequential tests](https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/).
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/sampling-distribution-of-the-difference-in-means
# Lesson 3: Sampling distribution of the difference-in-means estimator
> Using probability theory, we know how the difference-in-means
estimator varies across all possible samples and treatment assignments,
without going through every combination.
Using probability theory, we can calculate, without going through all random
samples and treatment assignments, how the difference-in-means estimator will
vary across all possible samples and treatment assignments. In fact, we even
know the precise distribution that the difference-in-means estimator will have
across all possible samples and treatment assignments. For means and difference
in means, the result that lets us do this is called the **Central Limit
Theorem**. The Central Limit Theorem states that if the sample size is large
enough, then the difference-in-means estimator will be normally distributed
around the true average treatment effect across random samples and treatment
assignments.
**Note:** The difference-in-means estimator is approximately normally distributed
around the true treatment effect, regardless of the distribution of the data. In
other words, even if data is not even remotely close to normally distributed,
the difference-in-means estimator will be normally distributed around the true
treatment effect if the sample size is large enough.
## You only observe a point estimate
Importantly, the observed difference in means in a given sample is not normally
distributed since it's just a fixed value. It is the difference-in-means
**estimator** across random samples and treatment assignments that is normally
distributed. This means that if you would run the experiment many times, the
difference in means you observe would be normally distributed.
## Simulation
In this simulation, we draw a random sample, split it
randomly into treatment and control, and calculate the difference in means. We
do this many times to see how the difference in means varies across random
samples and treatment assignments. Note that there is no treatment effect in
this simulation. The variation in the difference in means is only due to random
variation in the sample and treatment assignment. The observed distribution is
called the *sampling distribution of the difference-in-means estimator*, as it
is the distribution this estimator has across random samples and treatment
assignments.
[Interactive: Simulation]
The magic that probability theory and statistics bring us is that **we know the
what distribution will be a good approximation of the 500 simulated
difference-in-means estimates under the null before we have run the simulation.**
It works, because of math!
The value of knowing the distribution of the difference-in-means can't be
overstated. It lets us observe one sample and still draw conclusions (make
inference) about the full population. More on that in the next lesson.
**Quiz: According to the Central Limit Theorem, what is the shape of the sampling distribution for the difference-in-means estimator when the sample size is large enough?**
- Uniform distribution
- Normal distribution (correct)
- This cannot be known, it depends on the data
- A multimodal distribution
**Quiz: What exactly is normally distributed according to the lesson?**
- The observed difference in means from a single sample
- The true treatment effect in the population, regardless of the metric
- The difference-in-means estimator across many random samples and treatment assignments if the samples are large enough (correct)
- The values within each individual sample if the sample is large enough
**Quiz: In the simulation described in the lesson, what causes the variation in the difference-in-means estimates when there is no treatment effect?**
- Measurement errors in the data collection
- Changes in the treatment effect over time
- Random variation in both sampling and treatment assignment (correct)
- Differences in the population characteristics
## Notes for nerds
There are some technicalities in the Central Limit Theorem that we have glossed
over. The Central Limit Theorem states that the difference-in-means estimator is
normally distributed around the true treatment effect if the sample size is
large enough. The exact conditions for when the Central Limit Theorem holds are
a bit more nuanced, but for the purposes of this course, we can assume that the
Central Limit Theorem holds when the sample size is large enough. In principle,
as long as the underlying data doesn't have too fat tails, the Central Limit
Theorem will hold.
There are ways of making inference that is not based on the Central Limit
theorem. One example is the bootstrap method, which is a resampling method that
can be used to estimate the distribution of an estimator without making
assumptions about the distribution of the data. The bootstrap method is a
powerful tool that can be used in many situations where the Central Limit
Theorem doesn't hold. However, the bootstrap method is more computationally
intensive but there are some tricks to make it faster. See for example
[our blog post](https://engineering.atspotify.com/2022/03/comparing-quantiles-at-scale-in-online-a-b-testing/) on bootstrap for quantiles.
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/tpr-and-power
# Lesson 6: True positive rate, MDE, and power
> This page teaches you about the concept of true positive results in experiments. You learn:
- What a true positive result is.
- What the minimum detectable effect is.
- What the true positive rate is.
- What power is, and how it relates to the true positive rate.
## A true positive result
A true positive result, often simply called a "true positive", is when we find a
statistically significant effect from a treatment in an experiment, and the
treatment actually does have an effect. In other words, we correctly identify a
significant effect from the treatment, confirming that the treatment had a real
impact.
A true positive result is when we observe a large enough mean difference between
treatment and control groups to be statistically significant when the treatment
truly has an effect.
However, the fact that the treatment has an effect is no guarantee that we will
find a true positive result. By chance, people with large outcome values might
end up in the control group, and people with low outcome values might end up in
the treatment group. In such a situation, we might not find a true positive
result, even though the treatment has an effect, because the treatment is
canceled out by the difference between the groups caused by the random treatment
assignment.
[Interactive: UserAssignmentPowerSimulator]
Thanks to the random assignment of users to treatment and control, probability
theory lets us quantify how likely it is with any level of random imbalance
between the groups; this is precisely what statistical tests do.
Valid statistical tests quantify the variability of the test statistic under the
hypothesis of no treatment effect, which can be used to bound the rate at which
we get true positive results for certain hypothetical effects.
**Note:** If we fail to detect an existing effect, we call this a false negative result. When a false negative result occurs it can also be called a "Type II error".
Bounding the true positive rate to be above a certain intended power is the same as bounding the false negative rate to be below a certain level.
## True positive rate
We can never be certain about whether an observed result in an experiment is
true or not, due to randomness. For this reason, statistical tests are derived
to limit the rate of finding the wrong result across many experiments.
A good property for a statistical test to have is a high true positive rate. The
true positive rate is the rate at which the test correctly identifies a
significant effect when there is a true effect from the treatment.
For example, if we run 100 experiments where the treatment truly affects the
outcome metric, and we find that 80 of these experiments show a significant
effect, the proportion of experiments where we correctly find a significant
effect (80/100=80%) is the true positive rate of this test.
Having a high true positive rate means that the false negative result rate is
low, which is good because it means that we are not missing true effects.
## The minimum detectable effect (MDE)
How large the true positive rate is depends on several things, including the
false positive rate. Importantly, the true positive rate depends on the size of
the unknown but true treatment effect.
If the treatment effect is huge relative to the variability of the outcome
metric, the true positive rate will be high. If the treatment effect is tiny
(but not zero), the true positive rate will be low.
To derive statistical tests with bounded true positive rates, we use the concept
of the Minimum Detectable Effect (MDE). The MDE is the smallest effect size that
we want to be able to detect in an experiment with a certain true positive rate.
In other words, if there is a true effect of MDE, we want to be able to bound
the true positive rate higher than a certain rate.
## Power (the intended true positive rate)
Power is a parameter that statistical tests have that corresponds to the lower
bound on the true positive rate.
We say that a statistical test is powered for a certain effect (MDE) if the true
positive rate over repeated experiments (with a true effect of MDE) is higher
than or equal to a desired level of power.
In other words, by using powered statistical tests, we can bound the proportion
of experiments where there is a true effect that we fail to detect to `1-power`.
The value `1-power` is called the false negative rate and is often represented
by beta.
At this point, you might be wondering why we cannot simply set power to 100% to
find all true positive results. The reason is that this would require an
infinitely large sample. The relation here is: the larger the sample size the
smaller the standard error of the mean difference, and therefore the higher our
ability to detect small effects.
Risk management in experimentation is about balancing the risk of false
positives and chance for true positives against the sample size required to
achieve these risk bounds. Learn more about sample size and how to calculate the
required sample size for your experiment in the
[Sample size calculation - level I course](../sample-size-calculation-i/introduction).
For a fixed alpha, the higher power we want, the larger the sample size we need.
We can increase alpha to reduce the required sample size for a given level of
power and MDE, but this increases the false positive rate if in fact the
treatment doesn't have an effect.
For a fixed alpha and power, we can increase the MDE, but this means that we can
only find larger effects, and we might miss smaller effects even if they are
practically important.
**Quiz: What is a true positive result in an experiment?**
- When a statistically significant effect is found, but the treatment actually has no effect
- When the treatment has an effect, but the result is not statistically significant
- When a statistically significant effect is found, and the treatment actually has an effect (correct)
- When the null hypothesis is true, but we fail to reject it
**Quiz: What does the Minimum Detectable Effect (MDE) represent?**
- The smallest effect size we want to detect with a certain true positive rate (correct)
- The largest effect size that can be detected in an experiment
- The smallest effect size that guarantees no false positives
- The smallest sample size required to detect an effect
**Quiz: Why can't we simply set power to 100% to detect all true positive results?**
- Because it would require an infinitely large sample size (correct)
- Because it would reduce the false positive rate to zero
- Because it would increase the Minimum Detectable Effect (MDE)
- Because it would make statistical tests invalid
### Notes for Nerds
Sometimes the advice is given to "not trust underpowered experiments". This is
because significant treatment effects observed in underpowered experiments are
by construction over-estimated. See for example
[this paper](https://sites.stat.columbia.edu/gelman/research/published/retropower20.pdf) for
details.
However, this advice shouldn't be given without reference to the MDE for which
the experiment is powered, especially in relation to the true effect. What
really matters is the true positive rate, which is a function of the true
effect. If the experiment is underpowered for a very small MDE, but the true
effect is very large, the true positive rate might be high even when the
experiment is underpowered according to its MDE.
The bottom line is that the power is always in relation to a hypothetical effect
(MDE). The true effect can be anything, and thus the actual true positive rate
can be much larger or smaller than the intended power, even if our sample size
is sufficiently large to power the experiment for the MDE we have decided.
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/true-vs-estimated-effects
# Lesson 2: True versus estimated effects
> In an experiment, you don't observe the treatment effect in the full
population. You only observe a random sample of the population in your
experiment, and, the sample is randomly split into treatment and control.
In an experiment, you don't observe the treatment effect in the full population.
You only observe a random sample of the population in your experiment with that
sample randomly split into treatment and control. You only observe the mean of
the non-treated group for the subset of users in the sample that are in your
control group, and only observe the mean of the treated group for the subset of
users in the sample that are in your treatment group.
Clearly, the estimated difference in means between the treatment and control
group is not the same as the true treatment effect in the population. The
estimated difference in means varies depending on which users end up in your
random samples, and which of the users in the sample that end up in the
treatment and control groups.
In the illustration below, three random samples are drawn from a population. The
samples are split into treatment and control, and exposed to different variants
of a mobile app. The samples are small, which makes the estimates very
uncertain. In some samples, the estimated difference in means is larger than
zero, in some smaller than zero. This variation is referred to as the *sampling
variation* of the difference-in-means estimator: It's the
variation of the difference-in-means estimator across random samples and
treatment assignments.
[Image: Experimentation Flow Several Samples]
A treatment effect estimator is said to be **unbiased** if the average of all
estimates across all possible random samples and treatment assignments is equal
to the true population treatment effect.
## Separate the signal from the noise
So how do you know if the observed difference is due to random variation? Did
users with a high value of the outcome metric by chance end up in the treatment
group, or did the treatment actually have an effect? This is where statistics
comes in.
Because the sample and treatment assignment is random, probability theory lets
us quantify the uncertainty in the estimated difference in means under the null
hypothesis. If the treatment has no effect, then any variation in
difference-in-means estimates across random samples only occurs because different users
with different outcome values happen to be placed in different treatment groups.
**Note:** Probability theory lets us quantify how likely a certain mean difference
is if the treatment has no effect (meaning that the null hypothesis is true). If
the observed difference is very unlikely under the null hypothesis, we reject
the null hypothesis and conclude that the treatment has an effect.
The idea of *rejecting the null because the observed outcome is unlikely under
the null* can be challenging to digest. But this is important to understand to
build intuition for experimentation.
We say that a mean difference is **statistically significant** if it's among the
alpha percent most unlikely mean differences under the null hypothesis. If
that's the case, we *reject the null hypothesis* and say that "we found evidence
for the alternative hypothesis". Alpha is a parameter that the experimenter
sets, we return to alpha in [Lesson 5](./../hypothesis-testing/fpr-and-alpha).
The logic of rejecting the null is that since it would be much more likely to observe a large
mean difference if the treatment indeed had an effect, we rather believe that
the treatment has an effect than believe that the null hypothesis is true and
that we just observed a very unlikely mean difference by chance.
**Recommendation:** If this was your first time hearing about statistical
significance and rejecting the null hypothesis, don't worry. This is a concept
that takes time to understand. Go back to this lesson tomorrow or in a week an
do it again. For most new experimenters, this takes a few attempts to understand.
But how can we know if an observed mean difference is among the alpha percent
most unlikely mean differences under the null hypothesis? We can certainly not
go through all samples and treatment assignments and give everyone no treatment
(just in the example above there are more than 800 million combinations of
samples and treatment assignments). In the next section, we dig into how we can
know the distribution of the difference-in-means estimator under the null
hypothesis without going through all possible samples and treatment
assignments—using math.
**Quiz: Why might the estimated difference in means from an experiment differ from the true treatment effect in the population?**
- Because we only observe outcomes in a random sample, not the full population (correct)
- Because the treatment is inherently unstable and changes over time
- Because people always respond differently to treatments
- Because statistical analysis is imperfect
**Quiz: What causes sampling variation in the difference-in-means estimator?**
- Errors in measuring the outcome variable
- Different implementations of the treatment
- Random samples and random assignment of which users end up in treatment vs control groups (correct)
- Changes in the treatment effect over time
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/what-is-hypothesis-testing
# Lesson 1: Introduction to hypothesis tests
> This lesson introduces hypothesis testing by looking at the relation between the experiment hypothesis and the hypotheses in a hypothesis test.
You learn about the null and alternative hypotheses in a hypothesis test, and how they relate to the experiment hypothesis.
You also learn why you often test the mean of metrics in experiments, and how the randomness in the experiment helps us reason about the true average treatment effect.
## Experiment hypotheses
Most people are familiar with the concept of a hypothesis: it's a somewhat
formal statement about what you believe will be the outcome in an experiment. If
you have ever read some introduction to experimentation (including our
[introductory course](./../intro-course/experiment-hypothesis)), it's often
stated how important it is to have a clear hypothesis. It's worth noting that
**the hypothesis in an experiment is not the same as the hypotheses in an
hypothesis test.** The hypotheses in a hypothesis test are mathematically
precise statements about an aspect of a metric of interest. The test hypotheses
should of course reflect the experiment hypothesis, but they are not the same.
An example of a good _experiment hypothesis_ is:
**Based on** _user research,_ **we believe that** _having to create a username
creates friction in the sign-up process._ **We think that** _removing the step to
enter a username_ **for** _users signing up in the app_ **will lead to** _faster
completion of the sign-up flow._ **We will know this is true when we see** _a
decrease in the mean sign-up completion time_.
The last part about decreasing the mean sign-up completion time is the part we
test using hypothesis testing. To do that we need to translate the experiment
hypothesis into the two hypotheses in an hypothesis test.
## The null and alternative hypotheses of an hypothesis test
A hypothesis test has a null hypothesis and an alternative hypothesis.
Hypotheses are specific statements about aspects of a metric of interest, such
as the mean sign-up completion time that you want to improve. For example, a hypothesis
might say "we think the mean sign-up completion time will decrease by 10 seconds".
**Note:** An experiment hypothesis is often a less mathematically precise version
of the alternative hypothesis in a hypothesis test. In our example, "faster
completion of the sign-up flow" becomes a precise statement about decreasing mean
sign-up completion time.
Simply put:
- **The alternative hypothesis** describes what happens to the metric if the treatment has an effect (in our case, that removing the username field decreases mean sign-up completion time).
- **The null hypothesis** describes what happens to the metric if treatment has no effect (in our case, that removing the username field doesn't change mean sign-up completion time).
With these hypothesis in mind, we have a clear definition of what we expect will
happen with the metrics if the treatment has or has not an effect.
### Hypotheses for means
When talking about hypothesis tests in online A/B tests, we almost always talk
about a test of the difference in means between treatment and control. You have
an outcome metric of interest (like sign-up completion time), and this metric
has some mean before starting any experiment. As a product team, you are trying
to improve this metric by iterating on your product. When you are about to run
an experiment, you are interested in if a treatment (like removing the username
field) has a positive effect on the mean of this metric.
For our sign-up completion time example, we want the mean time to decrease after the treatment,
and the null and alternative hypotheses would be:
- $H_0$: Mean sign-up completion time in control group = Mean sign-up completion time in treatment group (null hypothesis)
- $H_1$: Mean sign-up completion time in control group > Mean sign-up completion time in treatment group (one-sided, negative direction)
This example shows how we can have directionality in our hypothesis test. We specifically
want to see a decrease in the mean time. In other contexts, where we might want to see
an increase in a metric (like mean revenue per user), we would adjust the alternative
hypothesis accordingly:
- $H_0$: Mean revenue in control group = Mean revenue in treatment group (null hypothesis)
- $H_1$: Mean revenue in control group < Mean revenue in treatment group (one-sided, positive direction)
**Note:** Because the
hypothesis is used in the test formulas, it's common to present them in a
mathematical form. For the purpose of this course, and to build intuition
about hypothesis testing, it suffices to understand them as precise statements
about some aspect of a metric.
There are always two hypotheses in a hypothesis test: the null hypothesis and
the alternative hypothesis. You can be mathematically fancy and write $H_0$ for
the null hypothesis and $H_1$ for the alternative.
## Why look at the mean?
You can test other parameters than the mean of the metric, but for
experimentation the mean is the most common. The mean is a good summary of a
metric to base decisions on, and, the mean has attractive statistical properties
that makes it easy to manage risks for using statistics.
The steps for running an experiment and testing the mean difference between
treatment and control are illustrated in the figure below. We:
1. take a random sample of users
2. split the sample randomly into treatment and control groups
3. give the treatment to the users in the treatment group
4. observe the mean of the metric in the two groups
5. compare the means using a hypothesis test.
[Image: Experimentation Flow]
## A Spotify example
For a more concrete example for Spotify, suppose the metric of interest is
audio book minutes played. The current mean is 48 minutes per user, and we are
interested in if a new version of a recommendation algorithm increases the mean
minutes played from 48 minutes to a higher number. In other words, is the new
version of the recommendation algorithm causing an increase in the average
minutes played? In an experiment, we test if the treatment affects the mean of
the metric by taking a random sample, splitting it randomly into treatment and
control, giving the treatment to the users in the treatment group, and then
observe the mean of the metric in the two groups and compare the means.
## How the randomness helps us manage risks
When the goal is to affect the mean, the hypothesis refers to the difference in
the means between the treatment and control groups. If the treatment has no
effect, the treatment and the control group should have the same mean and the
mean difference should be zero. If the treatment increases the mean, the
treatment group should have a higher mean than the control group, which would
lead to a mean difference larger than zero. The null hypothesis is that the
difference in means is zero. The alternative hypothesis is that the difference
in means is greater than zero.
Since the sample and treatment assignment is random and no person is exactly
like any other person, the difference in means between treatment and control
will not be exactly zero even if the treatment truly has no effect. The
difference-in-means estimator gives an estimate of the true average treatment
effect in the population, and it will vary across random samples and treatment
assignments. In the next lesson we will dig deeper into the variation of the
difference-in-means estimator across random samples and treatment assignments.
**Quiz: What is the primary purpose of hypothesis testing in experimentation?**
- To prove the experiment is correct
- To handle uncertainty and manage risk (correct)
- To increase the sample size
- To make the data look better
**Quiz: In a typical experiment, what do we usually test?**
- The difference in medians between treatment and control
- The difference in means between treatment and control (correct)
- The difference in variances between treatment and control
- The difference in modes between treatment and control
## Notes for nerds
It's common to call the population means in the treatment groups $μ_0$ and $μ_1$
to keep notation succinct. The null hypothesis is then $H_0: μ_0 = μ_1$ and the
alternative hypothesis is for example $H_1: μ_0 < μ_1$.
The hypotheses in a hypothesis test refer to so-called population parameters.
The population parameters are the true values of the metric in the population.
In other words, what would be the true average treatment effect if all users in
the population got this treatment. In an experiment, we can only observe the
metric in a sample of the population, and we use the sample mean as an estimate
of the population mean. The hypothesis test is then a test of the null
hypothesis that the population means are equal, based on the sample means.
Following lessons are devoted to unpacking the relations between the sample and
the population.
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/z-tests-and-rejecting-the-null
# Lesson 4: Z-tests and how to reject the null hypothesis
> Since we know that the distribution of the mean difference under the null
hypothesis is normally distributed with large samples, we can calculate how uncommon certain
differences in means would be under the null hypothesis.
**Note:** To simplify examples, **metrics are assumed to improve when they increase**. In all hypothesis tests in this lesson, the aim is to find evidence that the metric has increased as opposed to not moved at all.
Since we know that the distribution of the mean difference under the null
hypothesis is normally distributed with large samples, we can calculate how
unlikely a certain observed difference in means is under the null hypothesis. We
can see where on the normal distribution the observed difference lies and say
that it's among the x% most unlikely differences under the null hypothesis by
calculating the area in the tail of the distribution (which computers happily do
for us).
This is how hypothesis testing works: We calculate how
uncommon the observed difference in means is under the null hypothesis based on
the quantiles of the theoretically known sampling distribution. If the observed
difference is among the alpha % most unlikely difference in means you can
observe under the null hypothesis, we reject the null hypothesis.
## Z-tests
A Z-test is a statistical test used to determine whether there is a significant
difference between the means of two treatment groups used when the sample size
is sufficiently large for the Central Limit Theorem to make the sampling
distribution of the mean difference normally distributed.
The Z-test calculates the Z-score, which is a standardized value that tells us
how many standard deviations the observed difference in means is from the
expected value under the null hypothesis. This Z-score can then be used to
evaluate hypotheses in several ways.
For a Z-test, we use the following formula for the Z-score:
$Z_{obs} = \frac{\text{Observed statistic} - \text{Value under } H_0}{\text{Standard error of the statistic}}$
- Observed statistic: the sample mean (or difference in means) that we observe in our data.
- Value under ($H_0$): the value of the statistic we would expect under the null hypothesis. For example, in a two-sample test, the expected difference in means under the null hypothesis is usually 0.
- Standard error of the statistic: the description of the variability in the sampling distribution of the statistic under the null hypothesis. A smaller standard error means we are more confident in our estimate of the mean or difference in means.
When the Z-score ($Z_{obs}$) is calculated it can be used in three ways:
- To compare the $Z_{obs}$ to a critical value based on the significance level alpha (denoted $Z_{crit}$).
- To calculate a confidence interval around the point estimate, indicating how much uncertainty the estimate has.
- To determine the p-value by finding the probability of observing a Z-score as extreme as (or more extreme than) $Z_{obs}$ under the null hypothesis.
**Note:** The Z score ($Z_{obs}$) is the distance between the observed mean difference and the
value under $H_0$ in terms of number of standard errors.
## Critical Z values
One way to reject the null hypothesis in a Z-test is to compare the observed Z
score with a so-called critical Z value, denoted $Z_{crit}$. $Z_{crit}$ depends
on alpha, as it is the Z-value such that only alpha percent of the observed
statistics are larger than that across random sampling under the null
hypothesis.
For example, the $Z_{crit}$ for alpha 0.05 is 1.645, meaning that only 5% of the
observed difference-in-means are more than 1.645 standard errors from the null
hypothesis when the null hypothesis is true.
You reject the null hypothesis if $Z_{obs}>Z_{crit}$.
**Note:** The $Z_{crit}$ is the
number of standard errors that an observed statistic needs to be from the null
to be significantly different from the null.
## Confidence intervals
A one-sided lower bound confidence interval for difference-in-means point estimate is calculated as
$L = \text{Observed Statistic} - Z_{crit} \times\text{Standard error of the statistic}$.
Note that the theoretical interval is now between L and + infinity. It's common
practice to show two-sided confidence intervals, but for the purpose of building
intuition for hypothesis testing in online experimentation, we stick to the
one-sided.
This is called a $1-\alpha$ confidence interval. For example, if alpha is 5%, the we call it _a 95% confidence interval_.
The definition of a confidence interval is: _Across random samples and treatment
assignments, the confidence interval covers the true population treatment effect
at least $1-\alpha$% of the times._
With a one-sided confidence interval, you reject the null hypothesis if Lower bound > Value under
$H_0$. For example, if the difference-in-means is zero under $H_0$, you reject
the null hypothesis if $L>0$.
## P-values
You might have heard about p-values. They quantify: _How likely is it to obtain
the observed mean difference (or a larger difference) under the null hypothesis._
You reject the null hypothesis if the p-value is smaller than the selected alpha.
For example, a p-value of 0.055 means that the observed difference is among
the 5.5% most unlikely differences under the null hypothesis. If alpha is 10%
we reject the null since 5.5% is less than 10%. If alpha is instead 5% we fail
to reject the null since 5.5% is larger than 5%.
The smaller the alpha we use for the test, the more unlikely the observed
difference must be (so that the p-value is small enough) for us to reject the
null hypothesis.
## An interactive example
Use this playground to build intuition for how to reject the null using
confidence interval, the observed Z-score, and the p-value. It can also help
answer the questions below.
[Interactive: InteractivePlot]
## Let's summarize
Hypothesis testing is a mouthful the first time you dig into it. Before we move
on to the two last lessons, let's review what we've talked about.
Hypothesis tests let us quantify how likely or unlikely an observed
difference-in-means estimate is if the null hypothesis of a zero effect is true.
If the observed estimate is very unlikely under the null hypothesis, and much
more likely under the alternative hypothesis, we reject the null hypothesis and
conclude that the treatment has an effect.
To test if the mean of an outcome metric has improved due to a treatment, we
test if the difference in means between treatment and control is larger than
zero. For mean differences, we know the probability distribution of the
difference in means under the null hypothesis across random samples and
treatment assignments is the normal distribution. This lets us
calculate how unlikely a certain difference in means is under the null
hypothesis.
One thing that might bother you at this point is that, if we reject the null
hypothesis whenever we observe a mean difference that is more unlikely than alpha,
then we will reject a true null hypothesis in alpha % of all experiments where
there truly is no effect. Statistically significant really means "the difference is probably not
by chance", or "the difference is unlikely to be only due to random variation".
This is where risk management comes into the picture. Hypothesis testing cannot
help us reach the right conclusion in any given experiment with complete
certainty. It can only help us limit the risks across many experiments. More on
this in the following lessons.
**Quiz: What does a p-value represent?**
- The probability that the null hypothesis is true
- The probability that the alternative hypothesis is true
- The probability of making a Type I error
- The probability of observing a result as common/rare as the one observed, assuming the null hypothesis is true (correct)
**Quiz: When do we reject the null hypothesis?**
- When the p-value is less than alpha (correct)
- When the p-value is greater than alpha
- When the p-value is equal to alpha
- When the p-value is exactly 0.05
**Quiz: What is the relation between the observed z-score, the observed mean difference, and the observed standard error?**
- They are independent and only coincide by chance
- The z-score is the observed mean difference divided by the p-value
- The z-score is the mean difference divided by the standard error (correct)
---
Source: https://confidence.spotify.com/bootcamp/hypothesis-testing/course-wrap-up
# Course wrap up
**Congratulations! You have finished this primer on hypothesis testing!**
If this was your first course on hypothesis testing, you have learned a lot.
Hypothesis testing is not entirely intuitive for most people at first. We
recommend that you come back to the lessons of this course in a week or two to
reinforce your understanding.
Thank you so much for taking this course and helping to build an even stronger experimentation
community at your company!
Go back to [my learning page](..) to keep learning!
---
## Sample size calculation I
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-i/introduction
# Welcome to *Sample size calculation - level I*
[Interactive: CourseProgressBar]
*Sample Size Calculation — Level I* is an interactive, self-paced course that
teaches you the fundamental concepts of how to calculate the required sample
size for experiments. This course focuses on building intuition for why you need
to plan the sample size before the experiment and what settings and aspects that
affect the required sample size.
In this course, you'll:
- Get a short recap on alpha and power, and how they relate to sample size calculations
- Learn how the baseline mean and variance of a metric affect the required sample size
- Build intuition for sample size calculations buy working in an interactive sample size calculator playground
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material. Complete each lesson's questions to track your
progress.
## Before you begin
Before you start this course, you should go through the following lessons from
the hypothesis testing course:
- [Lesson 5: False Positive Rate and Alpha](./../hypothesis-testing/fpr-and-alpha)
- [Lesson 6: True Positive Rate, MDE, and Power](./../hypothesis-testing/tpr-and-power)
or take the full
[Hypothesis Testing course](./../hypothesis-testing/introduction).
## Lessons
This course consists of the following lessons. Click a lesson to start learning.
- Lesson 1: What is the required sample size?: Learn why it's crucial to calculate the required sample size before running an experiment.
- Lesson 2: Alpha and power: Revise your understanding of these fundamental concepts to ensure you can follow the rest of the course.
- Lesson 3: Baseline mean and variance: Understand how the baseline mean and variance of a metric affects the required sample size.
- Lesson 4: Sample size calculation playground: Build intuition for sample size calculation by playing with different settings in the playground.
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-i/alpha-and-power
# Lesson 2: A Refresher on Alpha and Power
> This lesson is a brief summary of lessons 5 and 6 from the
[Hypothesis Testing course](./../hypothesis-testing/introduction), to make sure you have what you
need to understand the sample size calculation course.
## Possible outcomes in experiments
In an experiment, there either exists a treatment effect or there doesn't, and
you either detect it or you don't. This gives us four possible outcomes depicted
below.
[Image: Possible outcomes]
Across many experiments, these four outcomes will occur with some rates. That
is, if we run 100 experiments, some number of them will end up in each of the
four quadrants.
[Image: Possible outcomes]
In hypothesis testing, **alpha is used as a parameter to control the rate of
false positive results** among the experiments that have no effect, and **power
is used to control the rate of true positive results** among the experiments
where there is an effect. **We call alpha and power the intended error rates** of
the test.
Our goal with experimentation is to control the rates of incorrect and correct
results. We can trade off between the rates of false positives and false
negatives by changing the **alpha**, **power**, and **sample size** of our test.
In fact, there are several things that affect the risk handling in experiments,
which we will cover in future courses. But for now, let's not get ahead of
ourselves.
By using a statistically valid test with a certain alpha, and a sample size
large enough for a certain MDE to achieve a certain power, we can:
- Bound the proportion of experiments without an effect that falsely detect an effect to be lower than or equal to alpha
- Bound the proportion of experiments with an effect of MDE (or larger) that correctly detect that effect to be larger than or equal to power.
---
## Video recap
If you haven't already, watch this 4-minute and 31-second video to quickly review what we've learned so far:
[Video: https://www.youtube.com/watch?v=u3yxBXzgmk4]
---
## Win rate across all experiments
Having powered tests does not bound the true positive rate across all
experiments you run. It only bounds the true positive rate for the subset of
experiments that have a true treatment effect of MDE or larger.
In practice, some experiments will have a non-zero effect smaller than the MDE
for which we have designed the test. In those experiments, our chance to detect
the treatment effect will be smaller than power.
The best we can do is to make sure that we select MDEs that map to the smallest
effect size that is practically relevant for our business. By powering all
experiments to detect that effect, we can ensure that our true positive rate is
at least power for all experiments in which the true effect is of a relevant
size.
---
## The nonlinearity of alpha and power
It is important to understand how the alpha and power parameters affect the
sample size. Because we, in most cases, use a Z-test for evaluating
experiments, a normal distribution underlies the dependency between required sample size, alpha and power. This means that **the
required sample size is not increasing linearly with alpha or power**. This
makes it much harder to reason about how the required sample size changes with
changes to the alpha or power.
### The Alpha z-value
In the sample size calculation, the alpha parameter comes into the equation via
a z-value. Although this is the same type of z-scores that we have discussed in
previous lessons, here let's not focus on the rationale for the z-value being in
the equation, but rather on the relation between alpha and the z-value.
The alpha enters into this sample size formula via a z-value because we are
using a Z-test that is based on the asymptotic normality of the
difference-in-means sample estimator. It is good to know that the relation
between alpha and z-alpha is nonlinear. This implies that changing the alpha by
a fixed value will change the required sample size by different amounts
depending on the alpha you had to begin with. Changing from 0.02 to 0.01 will
increase the required sample size more than changing from 0.1 to 0.09.
Note that the asymptotic normality that the Z-test (and therefore the sample
size calculations) is based on doesn't require the underlying data to be normally
distributed. Instead, it's the difference-in-means estimator that needs to be
approximately normally distributed under the null hypothesis, which it is for many
underlying data distributions thanks to the central limit theorem. Learn more
about the distribution of the difference-in-means estimator in the
[Hypothesis Testing course](./../hypothesis-testing/sampling-distribution-of-the-difference-in-means).
The plot below shows how the z-value changes with alpha.
[Interactive: AlphaZValuePlot]
### The power z-value
The same relation holds for how power comes into the sample size formula
[Interactive: PowerZValuePlot]
---
**Quiz: What is the primary goal of experimentation in terms of risks?**
- To maximize the number of true positives
- To control the rates of correct and incorrect results (correct)
- To eliminate false negatives entirely
- To increase the MDE for higher accuracy
**Quiz: What does a statistically valid test with a certain alpha guarantee?**
- The true positive rate across all experiments will equal alpha
- The false positive rate will be less than or equal to alpha (correct)
- The false negative rate will be zero
- The test will detect all effects, regardless of size
**Quiz: What happens in experiments where the true effect is smaller than the MDE?**
- The true positive rate will still equal the test's power
- The chance to detect the treatment effect will be smaller than power (correct)
- The false positive rate will increase beyond alpha
- The test will always detect the effect
**Quiz: What changes the required sample size the most (in absolute numbers), increasing alpha with one unit or decreasing it with one unit?**
- The absolute change is exactly the same
- Decreasing alpha increases the required sample size more (correct)
- Increasing alpha decreases the required sample size more
- Increasing alpha increases the required sample size more
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-i/mean-variance-and-mde
# Lesson 3: Baseline Mean Variance and the MDE
> This lesson teaches you about how the metric affects the sample size required
to power an experiment and what the Minimum Detectable Effect (MDE) is:
- The baseline variance of the outcome metric.
- The baseline mean of the outcome metric.
- What the Minimum Detectable Effect (MDE) and the relative MDE are.
---
## Baseline variance
The 'baseline variance', is just a fancy way of saying "the variance of the
outcome metric under no treatment". The baseline variance directly affects the
risk management of the experiment and therefore the sample size calculation.
As we will see in the following sections, the baseline variance goes directly
into the sample size calculation formula. If the variance increases with 10%, then the
sample size required to detect a certain MDE with a certain power also increases 10%.
---
## Baseline mean and the MDE
### Refresher on MDE
If you need a refresher on the concept of MDE, check out
[Lesson 6 in the hypothesis testing course](./../hypothesis-testing/tpr-and-power)
or watch this video:
[Video: https://www.youtube.com/watch?v=lV50V6hj2ms]
### Baseline mean
The baseline mean is just another word for "the average value of the outcome
metric under no treatment." The baseline mean is important because the relative
MDE is translated into an absolute MDE using the baseline mean.
[Image: Relation between relative and absolute MDE]
#### Baseline means close to zero
If the baseline mean is very small, then even a large relative MDE might
correspond to a practically irrelevant absolute MDE. This is particularly common
for binary metrics with very low rates.
Consider the following example:
The baseline mean of crash rates is **0.0001**. That is, one out of 10,000 users
experiences a crash on average. If we want to detect a **10% relative MDE**,
then the absolute MDE is **0.00001**. So we want to detect if the rate goes from
**0.0001 to 0.00011**, or, in other words, if one more user per 100,000 users
experiences a crash on average.
Since this is a very small change, we need a large sample size to detect it.
### Absolute versus Relative MDE
At this point you might wonder why, if we aren't aware of the baseline mean, why do we
use the relative MDE to express the effect we want to be able to detect?
This is the right question to ask! There are two reasons for why the relative
effect is often used. First, it is a nice way to understand impact in metrics
where the absolute values are hard to interpret or have intuition for like
minutes played at Spotify. Second, because of the first reason, most
experimentation tools let the user specify the MDE on a
relative scale. Which scale to use makes most sense is mainly a matter
of preference. In any case, understanding this relation and the importance of
always considering the baseline mean is helpful for all experimenters seeking to
understand sample size calculations.
---
**Quiz: How does the baseline variance of the outcome metric affect the sample size required for an experiment?**
- Higher baseline variance decreases the required sample size
- Baseline variance doesn\
- ,
(correct)
- ,
**Quiz: What happens if the baseline mean of the outcome metric is very small?**
- A large relative MDE might still correspond to a practically irrelevant absolute MDE (correct)
- The absolute MDE becomes irrelevant, regardless of the relative MDE
- The sample size required will always be small
- The baseline variance becomes more important than the baseline mean
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-i/ssc-playground-i
# Lesson 4: Sample size calculation playground - level I
> In this lesson, you build intuition for sample size calculation by using an interactive sample size calculator playground. By changing parameters that we have discussed in this course, you can see how the sample size changes.
This is the first level of the sample size courses, so a lot of the parameters
that you can affect in a real experiment are fixed here and cannot be changed.
The idea with this lesson is simple, you change the parameters in the sample
size calculator and see how the sample size changes. In this lessons there are
more questions than for usual lessons. Most can be answered by simply trying
things out in the sample size calculator.
**Recommendation:** Turn on the 'Show detailed formulas' option in the sample size calculator to
see the formulas used to calculate the sample size. This can help you
understand how the different parameters affect the sample size.
[Interactive: SampleSizeCalculator]
**Quiz: If you double the relative MDE/NIM (from 1% to 2%), what happens to the required sample size?**
- The sample increases
- The sample size stays the same
- The sample size decreases (correct)
- The baseline mean and variance become more similar
**Quiz: What happens to the required sample size if you increase the baseline variance while keeping all other parameters constant?**
- The sample size decreases proportionally
- The sample size increases proportionally (correct)
- The sample size remains unchanged
- The sample size increases exponentially
**Quiz: If you want to increase the power from 0.8 to 0.9, what happens to the required sample size?**
- It decreases slightly
- It increases substantially (correct)
- It remains approximately the same
- It doubles exactly
**Quiz: If power is 80% and alpha is 10%, what increases the sample size the most?**
- Increasing power to 90%
- Decreasing alpha to 5%
- Increasing power to 85 and alpha to 7%
- Increasing power to 95% and keeping alpha at 10% (correct)
**Quiz: If your baseline variance is very small (near zero), what tends to happen to the required sample size?**
- It becomes very large
- It becomes very small (correct)
- It stays moderate
## Notes for Nerds
**In Confidence:** The Confidence sample size calculator takes into account a whole range of parameters when calculating the required sample size. If you want to learn about how Confidence handles risk management, you can read more about it in [this blog post](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics/).
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-i/what-is-the-required-sample-size
# What is the required sample size?
> In this lesson you learn what the required sample size is, and why it's important to calculate it before running an experiment.
## What is the required sample size?
When you run an experiment, you can either collect too little data or too much
data, by running your experiment on too few or to many of your users. The number
of users in your experiment is called the _sample size_ of the experiment.
Having a too small sample can lead to inconclusive results, while
having a too large sample means wasting resources. **The required sample size
is the number of users you need in your experiment to achieve a certain level of
precision in the results of your experiment.**
It might be clear enough why it's bad to run experiments with unnecessarily
large samples, but what exactly does _inconclusive results_ mean? Inconclusive
results are when there is no significant effect of the change you made in the
results but there is so much uncertainty in the results that you can't say for
sure if there is an effect or not: Perhaps you missed an existing effect due to
all the noise in the data.
One way to think about the required sample size is to think about it as _the
sample size you need to reach a certain level of precision in your effect
estimates_. If you have a large sample, you will have high precision (tight
confidence intervals), if you have a small sample, you will have low precision
(wide confidence intervals).
How do you know what the perfect amount of certainty is? You find it by deciding
on a set of parameters that bounds the risk of reaching the incorrect conclusion
from the experiment. The following lessons will teach you about what those
parameters are and how, once they are specified, they determine the required
sample size. You decide what risks of incorrect conclusions your are willing to
take, and then calculate the required sample size to reach that level of
precision.
## Why do you need to calculate the required sample size?
It's possible to calculate the current powered effect during an experiment. This
makes it possible to stop an experiment when the sample size is just right.
However, there are several reasons why it's better to calculate the required
sample size before running the experiment.
If you don't calculate the required sample size before running an experiment, you don't know
if the sample size you _can_ reach is sufficient to reach the level of precision
you need to make a decision based on the experiment.
This is an important realization: The required sample size calculation tells you
what sample size you _need_ to reach a certain precision. That doesn't mean that
you _can_ reach that sample size. For example, it's not uncommon that you select
the risk parameters such that the required sample size is larger than the total
number of users that your product has. This is frustrating, as this means that
you can't learn what you'd wish with the precision you'd hoped for. But at least the
required sample size calculation lets you know that before you start the
experiment—which means that you don't have to waste time and resources going after
the impossible.
The other aspect of calculating the required sample size before starting the
experiment is that it helps you plan the experiment. In many cases, you can
change aspects of an experiment to require a smaller sample size. For example,
you can change the duration of the experiment, the number of variations you
test, or the level of risk you are willing to take. By knowing the required
sample size, you can make these decisions in an informed way.
**Quiz: What does 'inconclusive results' mean in the context of an experiment?**
- The experiment found a statistically significant effect.
- The experiment proved that the treatment had no effect.
- The experiment had insufficient data to confidently determine the presence or absence of a treatment effect. (correct)
- The experiment suffered from a high false positive rate.
**Quiz: Why is it crucial to calculate the required sample size before running an experiment?**
- To avoid collecting too much data and wasting resources.
- To ensure the experiment reaches a desired level of precision for decision-making.
- To determine the appropriate duration of the experiment.
- All of the above. (correct)
**Quiz: What does it mean if the calculated required sample size is larger than the total number of users for a product?**
- The experiment is perfectly planned and will yield the desired results.
- The experiment should be run with a smaller sample size to save resources.
- The desired level of precision might not be achievable with the available user base. (correct)
## Notes for nerds
It's not obvious that calculating the powered effect as a stopping rule during
an experiment is fine from an statistical inference perspective. Calculating the
current powered effect requires peeking at the data, which could lead to
inference issues, just like peeking at the data to stop the experiment when the
p-value is low.
However, looking at the current effect mean only 'peeking' at the sample
variance, which turns out to be fine. See
[our paper on precision based experimental designs](https://arxiv.org/pdf/2405.03487)
for more information.
---
## Sample size calculation II
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/introduction
# Welcome to *Sample size calculation — level II*
[Interactive: CourseProgressBar]
*Sample size calculation — level II* is an interactive, self-paced
course that teaches you the fundamental concepts of how to calculate the
required sample size for experiments.
In this second course, you will build on the knowledge you gained in the first
course and learn how to calculate the sample size for more complex experiments.
You will learn how to combine the results from many metrics into one decision.
You will learn how the number of success metrics, guardrail metrics, and number
of comparisons affect the sample size.
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material. Complete each lesson's questions to track your
progress.
## Before you begin
This is the second level of the sample size calculation course. If you haven't
already, start with
[Sample size calculation - level I](../sample-size-calculation-i/introduction)
to build a solid foundation in hypothesis testing and sample size calculation.
## Lessons
This course consists of the following lessons:
- Lesson 1: Multi-metric decision making: Learn how to combine the results from many metrics into one decision while managing risks.
- Lesson 2: Number of success metrics: Build intuition for how the number of success metrics affects the sample size.
- Lesson 3: Number of guardrail metrics: Understand why guardrail metrics affect the sample size different from success metrics.
- Lesson 4: Number of comparisons: Learn how the number of treatment groups and comparisons affects the sample size.
- Lesson 5: Sample size calculation playground - Level II: Build intuition for sample size calculation by playing with different settings in the playground.
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/multi-metric-decisions
# Lesson 1: Multi-metric decision making
> This lesson teaches you how to formalize decision-making from experiments with
guardrail and success metrics. Confidence uses a decision rule to map the results of
all success and guardrail metrics to one decision: Ship or not.
## The two main types of metrics
There are two types of metrics used in experiments: **success metrics** and
**guardrail metrics**. Guardrail metrics are metrics that we ensure don't move
in the wrong direction due to our product change. Success metrics are metrics
that we want to improve. By combining these metrics, we can make better, more
precise, product decisions.
## Spotify's decision rule
**At Spotify we use the decision rule:**
Ship if, and only if, **at least one success metric** has significantly
improved, and the treatment is significantly non-inferior to control in **all
guardrail metrics**.
This is also the default decision rule for all recommendations in Confidence.
The goal of this experimental design is to manage the risks associated with
decisions made using this rule. This means accounting for all metrics
and how they affect the decision rule - simultaneously.
In the following lessons, we explain how the number of success and guardrail
metrics affects the sample size calculation. If you want to learn more about how
Confidence manages risk in decision-making, read more in
[this blog post](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics/).
## Intro to guardrail metrics and non-inferiority tests
Guardrail metrics are different from success metrics. They are metrics that we
want to ensure do not significantly worsen in the treatment group compared to
the control group.
This means that we aim to prove that the metric did not deteriorate in the treatment
group compared to the control group. To do this, we use **non-inferiority
tests**.
This video introduces the concept of non-inferiority tests and how the choice of the Non-Inferiority Margin (NIM) affects the sample size calculation.
[Video: https://www.youtube.com/watch?v=1O3Hm-0fwZg]
There are more ways to use guardrail metrics in Confidence besides with
non-inferiority tests. Read more in
[this blog post](https://confidence.spotify.com/blog/better-decisions-with-guardrails).
**Quiz: What is the primary purpose of guardrail metrics in experiments?**
- To measure the success of a product change
- To ensure metrics do not move in the wrong direction due to a product change (correct)
- To increase the statistical significance of success metrics
- To calculate the required sample size for an experiment
**Quiz: What is Spotify's decision rule for shipping a product change?**
- Ship if at least one success metric has improved significantly
- Ship if all guardrail metrics have moved in the desired direction
- Ship if at least one success metric has improved significantly, and all guardrail metrics are significantly non-inferior (correct)
- Ship if the treatment group outperforms the control group in all metrics
**Quiz: What is the relationship between the Non-Inferiority Margin (NIM) and sample size?**
- A smaller NIM requires a larger sample size (correct)
- A larger NIM requires a larger sample size
- NIM doesn't affect sample size
- Sample size is only determined by success metrics
## Notes for nerds
Deterioration metrics and health checks also affect the risk management of decision-making. We don't want to ship if:
- Any metric included in the experiment moves significantly in the wrong direction.
- Any health check, like the **sample ratio mismatch test**, triggers.
These are also based on statistical tests and therefore have a risk of incorrectly triggering.
Read more in
[this blog post](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics/)
and in
[this academic paper](https://arxiv.org/abs/2402.11609).
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/number-of-comparisons
# Lesson 4: Number of comparisons
> This lesson teaches you about how the number of comparisons affects the
required sample size in experiments. Everything else held constant, the more
treatment groups and thereby comparisons you have, the smaller alpha you will
have to use per metric to bound the false positive rate for the decision below
alpha, which leads to a larger required sample size. The number of comparisons
does not affect the power you need to use per metric.
## The impact of multiple comparisons
The most common pattern in product A/B tests is to compare all treatment groups
against a control group. This means there are as many comparisons as there are
treatment groups being tested.
In principle, it is also possible to compare all treatment groups against each
other. This would mean the number of comparisons equals the number of pairs of
treatments.
The number of comparisons affects the required sample size. The more
comparisons, the more samples are required. This is because the probability of
making a **Type I error** (false positive) increases with the number of
comparisons. To counter this, we adjust the alpha level for multiple
comparisons, which increases the required sample size.
The intuition behind this adjustment is that the more tests we run, the more
chances there are to find a significant result by random chance. For example, if
we run an experiment with 100 treatments and an alpha of 10%, even if no
treatment has any effect, we would expect to see 10 treatments with a (false
positive) significant result just by chance.
**Quiz: How does the number of comparisons in an experiment affect the required sample size?**
- The required sample size decreases with more comparisons
- The required sample size increases with more comparisons (correct)
- The number of comparisons doesn't affect the required sample size
- The required sample size depends only on the power setting
**Quiz: Why is the alpha level adjusted when there are multiple comparisons in an experiment?**
- To increase the probability of detecting true positives
- To prevent the required sample size from decreasing
- To bound the false positive rate for the overall decision below alpha (correct)
- To ensure all treatment groups are compared equally
## Notes for nerds
Some people wonder what to do if more than one treatment is significantly better
than the control group. This is a deep question. You can test the
treatments against each other to see if one is better than the other. However,
the difference between the treatment groups is likely smaller than the
difference between them and the control group. This makes the power to detect a
difference between treatments lower than the power to detect a difference
between a treatment and the control group.
There are more advanced methods for finding the best treatment among many, such
as Tukey's and Scheffé's methods, and Dunnett's test. However, we
don't recommend using these methods due to the complexity involved in learning
how to use them. Instead, you should gather stakeholders to decide which of the significant
treatments to implement based on factors such as:
- Complexity
- Cost
- Future extensibility
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/number-of-guardrail-metrics
# Lesson 3: Number of guardrail metrics
> This lesson teaches you how the number of guardrail metrics affects the
required sample size in experiments. Only guardrail metrics with NIMs affect
the sample size: the more you have, the larger power you will have to use per
metric to bound the true positive rate above power, which leads to a larger
required sample size. Guardrail metrics without NIMs have no cost to the
shipping decision's sample size.
## Guardrail metrics with and without NIMs
Not all guardrail metrics have a Non-Inferiority Margin (NIM). A guardrail
metric **with a NIM** requires the experiment to pass a non-inferiority test
before shipping. A guardrail metric **without a NIM** is tested only for
regression (whether it has significantly deteriorated) and does not need to pass
a non-inferiority test for the experiment to ship.
**Adding a guardrail metric without a NIM has no cost to the shipping
decision's sample size.** It does not affect the alpha correction (it cannot
increase the shipping false positive rate) and does not affect the beta
correction (it cannot block the ship by not being significant).
There is, however, an alpha cost for the abort decision. Regression
tests across all user-specified metrics,
[required metrics](https://confidence.spotify.com/docs/surfaces#required-metrics), and health
checks like the
[sample ratio mismatch test](https://confidence.spotify.com/docs/monitoring) all contribute to the
alpha correction for abort. Adding many guardrail metrics without NIMs inflates
this correction, reducing the power to detect actual regressions. We don't
currently target a specific power for the abort decision, so this effect is not
tracked.
**The rest of this lesson applies to guardrail metrics with NIMs.** Read more
about how guardrail metrics with and without NIMs fit into smaller-sample
experimentation in
[Experiments with Smaller Samples](https://confidence.spotify.com/blog/smaller-sample-experiments).
## False positive rate
Since all guardrail metrics with NIMs must be simultaneously non-inferior, the
probability of a false positive for the decision rule decreases as the number of
guardrail metrics increases.
You can think about this as the difference between having 5 dice and needing at
least one six versus needing all 5 dice to show sixes. The more dice you have,
the less likely it is that all of them are sixes at the same time by chance.
However, if the guardrail metrics are highly correlated, the probability of
getting all sixes doesn't decrease as quickly with the number of dice.
**For this reason, we don't need to adjust alpha for the number of guardrail
metrics.**
## True positive rate
The power of the experiment is affected by the number of guardrail metrics.
Since we need all guardrail metrics to be non-inferior simultaneously, it is not
enough for each metric to have the probability power to be significantly
non-inferior under the alternative hypothesis.
[Interactive: GuardrailMetricsCalculator]
The plot above assumes that the metrics are independent. If they are highly
correlated, the probability of all metrics being significant doesn't decrease
as quickly with the number of metrics. However, without knowing the correlation,
we must ensure that the power is high enough in the worst-case scenario of
independent metrics.
**Quiz: How does the number of guardrail metrics with NIMs affect the required sample size?**
- The required sample size decreases with more guardrail metrics with NIMs
- The required sample size increases with more guardrail metrics with NIMs (correct)
- The number of guardrail metrics with NIMs has no effect on the required sample size
- The required sample size only depends on the alpha setting
**Quiz: Why does the probability of a false positive decision decrease as the number of guardrail metrics increases?**
- Because all guardrail metrics must be non-inferior simultaneously (correct)
- Because alpha is adjusted for the number of guardrail metrics
- Because guardrail metrics are always independent
- Because the power of the experiment decreases with more metrics
**Quiz: Why is it unnecessary to adjust alpha for the number of guardrail metrics?**
- Because the required sample size compensates for the number of metrics
- Because the true positive rate is unaffected by the number of metrics
- Because the probability of a false positive decreases naturally as the number of guardrail metrics increases (correct)
- Because guardrail metrics are highly correlated
**Quiz: How does the number of guardrail metrics affect the power of the decision of an experiment?**
- The power increases with more guardrail metrics
- The power decreases with more guardrail metrics (correct)
- The power remains unaffected by the number of guardrail metrics
- The power depends only on the correlation between metrics
**Quiz: What happens to the simultaneous power of the overall decision if the power correction is used and the guardrail metrics are highly correlated?**
- The decision is underpowered
- The power is unaffected
- The decision is overpowered (correct)
- The alpha needs additional adjustment for the results to make sense
**Quiz: What is the effect of adding a guardrail metric without a NIM on the required sample size for the shipping decision?**
- It increases the required sample size because it adds a regression test
- It increases the required sample size because it affects the beta correction
- It has no effect on the required sample size for the shipping decision (correct)
- It decreases the required sample size by reducing the false positive rate
## Notes for nerds
- **False positive rate:** For guardrail metrics, the false positive rate decreases as the number of metrics increases. This is because all metrics must be simultaneously non-inferior, making it harder to have a false positive as metrics are added.
- **True positive rate:** For true positives, the worst-case scenario is independent metrics, where each metric is treated as a separate hurdle. This decreases the overall probability of all metrics being significant.
If we knew the correlation structure between metrics, we could adjust the power
to account for dependencies, but this adds complexity to the design and
interpretation.
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/number-of-success-metrics
# Lesson 2: Number of success metrics
> This lesson teaches you how the number of success metrics affects the
required sample size in experiments. The more
success metrics you have, the smaller alpha you will have to use per metric to
bound the overall false positive rate for the decision below alpha. This leads to a
larger required sample size. The number of success metrics doesn't affect the
power you need to use per metric.
---
## False positive rate
Recall Spotify's multi-metric decision rule:
**Ship if, and only if, at least one success metric has significantly improved,
and the treatment is significantly non-inferior to control in all guardrail
metrics.**
Each additional success metric in the experiment gives an additional chance of
reaching the conclusion to ship the treatment. If each success metric is tested
with a significance level of 5%, the probability of having at least one false
positive increases with the number of success metrics.
To counter this, we adjust the significance level for each success metric test
to keep the chance of at least one false positive at 5% or smaller. The simplest
way to do this is the **Bonferroni correction**, where alpha is adjusted by
dividing it by the number of success metrics.
[Interactive: SignificanceCalculator]
As we can see, the Bonferroni correction effectively bounds the false positive
rate to be alpha% or smaller, regardless of the number of success metrics.
---
## True positive rate
The power of the decision rule is also affected by the number of success
metrics. However, instead of reducing the true positive rate, it **increases
it**.
If each success metric is powered with 80% power, the probability of having at
least one success metric with a true positive rate of 80% or larger increases
with the number of success metrics. This means no adjustment to the power
setting per success metric is needed to obtain the overall power for the
decision rule that is at least as high as the desired level of power.
---
**Quiz: How does the number of success metrics affect the required sample size in experiments?**
- The required sample size decreases with more success metrics
- The required sample size increases with more success metrics (correct)
- The number of success metrics has no effect on the required sample size
- The required sample size only depends on the power setting
**Quiz: What is the purpose of adjusting alpha for each success metric in experiments?**
- To ensure the power of each success metric remains high
- To bound the overall false positive rate for the decision below alpha (correct)
- To reduce the required sample size for the experiment
- To increase the probability of detecting a true positive
**Quiz: What is the simplest method to adjust alpha for multiple success metrics?**
- Bonferroni correction (correct)
- Non-inferiority tests
- Two-sided tests
- Bayesian adjustment
---
## Notes for nerds
1. **False positive rate:**
- For the false positive rate, independent metrics represent the worst-case scenario. Each metric creates a completely new chance for a false positive, necessitating alpha adjustment.
2. **True positive rate:**
- For the true positive rate, the worst-case scenario is that the metrics are perfectly dependent. In this case, only one chance exists for a true positive.
If we knew the correlation structure between metrics or made assumptions about it, we could reduce the conservativeness of the alpha adjustment and decrease the power per success metric. However, such methods add complexity in understanding and interpretation.
---
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/ssc-playground-ii
# Lesson 5: Sample size calculation playground - level II
> In this lesson, you build intuition for sample size calculations by using an
interactive sample size calculator playground. By changing parameters that we
have discussed in this course, you can see how the sample size changes.
This is the second level of the sample size calculation course, so there are still parameters that you can affect in a
real experiment that are fixed here and cannot be changed.
The idea with this lesson is simple, you change the parameters in the sample size calculator and see how the sample size changes.
In this lessons there are more questions than for usual lessons, and can be answered by trying things out with the sample size calculator.
**Recommendation:** Turn on the 'Show detailed formulas' option in the sample size calculator to
see the formulas used to calculate the sample size. This can help you
understand how the different parameters affect the sample size mathematically.
[Interactive: SampleSizeCalculator]
**Quiz: Does increasing the number of guardrail metrics with NIMs always increase the required sample size more than increasing the number of success metrics?**
- Yes, this is because guardrail metrics with NIMs increase the false positive rate.
- Yes, this is because the formula makes the sample size increase exponentially with the number of guardrails.
- No, because of the non-linearity of power and alpha in the formulas, which type of metric costs the most depends on the alpha and power settings. (correct)
- This question doesn\
**Quiz: How do increasing the number of success metrics and increasing the number of comparisons affect the sample size?**
- Increasing success metrics has a larger effect than increasing comparisons
- Increasing comparisons has no effect on sample size
- They both increase the required sample size at the same rate (correct)
- The effects depend on the correlation between metrics
---
## Sample size calculation III
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/introduction
# Welcome to *Sample size calculation — level III*
[Interactive: CourseProgressBar]
*Sample size calculation — level III* is an asynchronous, interactive,
self-paced course that teaches you the fundamental concepts of how to calculate
the required sample size for experiments. This course focuses on building
intuition for why you need to plan the sample size before the experiment and
what settings and aspects that affect the required sample size.
In this course, you'll learn about some of the more mathematical aspects of
sample size calculation. You learn about variance reduction and how it reduces
the required sample size. You also get to build intuition for how the sample
size calculation differs between binary and continuous metrics, and how the
treatment group proportions affect the required sample size. Finally, you
get to play around with the complete sample size calculator, accounting for all
the aspects covered in all three sample size courses.
**Note:** There are quiz questions throughout the course to help you check your
understanding of the material. Complete each lesson's questions to track your
progress.
## Before you begin
This is the third and final level of the Sample Size Calculation course series. If you haven't already, complete the previous levels before starting this course:
- [Sample size calculation - level I](../sample-size-calculation-i/introduction)
- [Sample size calculation - level II](../sample-size-calculation-ii/introduction)
## Lessons
This course consists of the following lessons:
- Lesson 1: Binary metrics: Understand how sample size calculations differ between binary and continuous metrics.
- Lesson 2: Treatment group proportions: Build intuition for why the relative sizes of the treatment groups affect the required sample size.
- Lesson 3: Variance Reduction: Learn why variance reduction might as well be called sample size reduction.
- Lesson 4: Sample size playground: Build intuition for sample size calculation by playing with different settings in the playground.
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/binary-metrics
# Lesson 1: Binary metrics
> This lesson explains how binary metrics differ from continuous metrics in
sample size calculations. For binary metrics, the variance under the alternative is know, this is not the case for continuous metrics.
---
## Variance of a binary metric
Binary metrics, such as whether a user clicked a button or made a purchase, have
specific properties. The variance of a binary metric is a deterministic function
of the mean (proportion of ones) of the metric.
Let `p` represent the proportion of ones in the metric. Then the variance of the
metric is calculated as **Variance = p \* (1 - p)**.
For example, assume we have a metric measuring whether a user clicked a button. The mean of the metric is the proportion of users who clicked the button.
- If the mean is `p = 0.5`, the variance is **0.5 \* (1 - 0.5) = 0.25**
- If the mean is `p = 0.1`, the variance is **0.1 \* (1 - 0.1) = 0.09**
---
## Variance in the treatment group
For continuous metrics, the best guess of the variance in the treatment group is
typically the same as the variance in the control group. This is not because we
might expect the variance to be the same, but because it is hard to make an
informed guess about how a treatment effect of size MDE (Minimum Detectable
Effect) would affect the variance.
For binary metrics, the situation is different. Since the variance of a
binary metric is a function of its mean, we can determine the variance in the
treatment group under a treatment effect of size MDE. Let `p` represent the proportion of ones in the control group. Then the
variance in the treatment group is calculated as **Treatment Variance = (p +
MDE) \* (1 - (p + MDE))**.
For example, if `p = 0.3` and the MDE is `0.1`, then the treatment group
variance would be **(0.3 + 0.1) * (1 - (0.3 + 0.1)) = 0.4 * 0.6 = 0.24**
---
**Quiz: How is the variance of a binary metric calculated?**
- Variance = p
- Variance = (1 - p)
- Variance = p * (1 - p) (correct)
- Variance = (p + MDE) * (1 - (p + MDE))
**Quiz: How does the calculation of variance differ between binary and continuous metrics?**
- For binary metrics, variance is a deterministic function of the mean, whereas for continuous metrics, variance depends one more than its mean. (correct)
- For continuous metrics, variance is a deterministic function of the mean, whereas for binary metrics, variance is always unknown
- Variance is calculated the same way for both binary and continuous metrics
- Binary metrics do not have variance, whereas continuous metrics do
**Quiz: How is the variance in the treatment group for a binary metric determined under a treatment effect of size MDE?**
- Variance = p * (1 - p)
- Variance = (p + MDE) * (1 - (p + MDE)) (correct)
- Variance = (p + MDE)
- Variance = p + (1 - p)
---
## Note for nerds
For **guardrail metrics**, the alternative hypothesis used in sample size
calculations assumes that the proportion in the treatment group is the same as
in the control group.
[Image: Variance of Binary Metrics]
Technically, the variance in the treatment group used in the sample size
calculation should therefore vary depending on whether the metric is a
**guardrail metric** or a **success metric**.
In practice, the difference is often small enough to ignore, but Confidence's sample size
calculator corrects for this.
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/ssc-playground-iii
# Lesson 4: Sample size calculation playground - level III
You have made it to the final boss. In this lesson, you get to play around with
**all** the parameters that affect the sample size calculation in our
calculator to get a feel for how the different parameters affect the sample
size.
> This is the third and final level, so now you have all the parameters from all three levels of this course free for you to play around with!
The idea with this lesson is simple: you change the parameters in the sample size calculator and see how the sample size changes.
In this lesson, there are more questions than in other lessons, and they can be answered by experimenting with the sample size calculator.
**Recommendation:** Turn on the 'Show detailed formulas' option in the sample size calculator to
see the formulas used to calculate the sample size. This can help you
understand how the different parameters affect the sample size.
[Interactive: SampleSizeCalculator]
---
**Quiz: What happens to the required sample size when you increase the baseline mean for a binary metric from 0.2 to 0.8?**
- The sample size decreases (correct)
- The sample size increases
- The sample size remains the same
- The sample size becomes invalid
**Quiz: How does reducing the Relative MDE/NIM (%) from 5% to 1% affect the required sample size?**
- The sample size remains constant
- The sample size is not affected by Relative MDE/NIM
- The sample size increases (correct)
- The sample size decreases
**Quiz: How does the required sample size differ between a binary metric with a baseline mean of 0.5 and a continuous metric with a baseline mean of 0.5?**
- It depends on the variance of the continuous metrics (correct)
- The sample size is larger for the binary metric
- The sample size is larger for the continuous metric
- The sample size is the same for both
**Quiz: What is the effect of increasing the Variance Reduction Factor from 0% to 50% on the required sample size?**
- The sample size remains constant
- The sample size decreases (correct)
- The sample size doubles
- The sample size increases
**Quiz: How does reducing the alpha (false positive rate) from 0.1 to 0.01 impact the required sample size?**
- The alpha level doesn't affect sample size
- The sample size increases (correct)
- The sample size remains unchanged
- The sample size decreases
**Quiz: What happens to the required sample size when increasing the power (true positive rate) from 80% to 95%?**
- The sample size increases (correct)
- Power does not affect the sample size
- The sample size remains constant
- The sample size decreases
**Quiz: How does increasing the number of comparisons from 1 to 5 impact the required sample size?**
- The sample size remains unchanged
- The sample size decreases
- The number of comparisons doesn't affect sample size
- The sample size increases (correct)
**Quiz: What happens to the required sample size and group allocation when you change the control group proportion from 50% to 80%?**
- The sample size remains constant, but the group allocation changes
- The sample size and group allocation remain unchanged
- The sample size increases relative to 50%/50% allocation (correct)
- The sample size decreases relative to 50%/50% allocation
**Quiz: How does increasing the number of success metrics from 1 to 3 affect the required sample size?**
- The number of success metrics doesn\'t affect sample size
- The sample size remains constant
- The sample size decreases
- The sample size increases (correct)
**Quiz: What is the effect of increasing the number of guardrail metrics with NIMs from 1 to 5 on the required sample size?**
- The sample size increases (correct)
- The sample size remains constant
- Guardrail metrics do not affect sample size regardless of NIMs
- The sample size decreases
**Quiz: What is the effect of increasing the number of guardrail metrics without NIMs from 0 to 5 on the required sample size for the shipping decision?**
- The sample size increases because more regression tests require more power
- The sample size increases because the alpha correction applies to all guardrail metrics
- The sample size remains unchanged (correct)
- The sample size decreases because regression tests reduce the false positive rate
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/treatment-proportion
# Lesson 2: Treatment group proportions
> This lesson explains how the relative sizes of treatment groups affect the
required sample size in experiments. If the total sample size is fixed, it's in most cases optimal to have equal group sizes. However, a larger total sample size is always better.
---
## Group size and power
When discussing required sample size, it is common to refer to a single number:
"the sample size." However, this total sample size is actually a combination of
the sample size in the control group and the sample size in the treatment group.
Interestingly, the total required sample size is not fixed if we change the
relative sizes of the treatment groups. This is intuitive: imagine you have a
sample of 100 users. When do you learn the most about the treatment effect? If
the groups are split 50/50 or 99/1? If only one user is in the treatment group,
it will not provide much information about the treatment effect.
---
## Fixed sample size: Equal group sizes maximize power
For **continuous metrics**, it is optimal from a power perspective to have
**equal group sizes**. For **binary metrics**, the optimal group sizes depend on
the Minimum Detectable Effect (MDE) and the baseline proportion. For the optimal
group sizes to deviate from equal, the baseline proportion and the proportion
under the hypothetical treatment effect must differ a lot. In other words,
unless the MDE is very large, it is a good general rule to aim for similar group
sizes. If you are interested in the mathematical details, see the derivation of
optimal treatment group sizes in the [Note for nerds](#note-for-nerds) section below.
**Recommendation:** If the total possible sample size is fixed, it is a good general rule to aim
for similar group sizes.
---
## Larger total sample size is always better
It's important to realize that:
1. For a **fixed total sample size**, it is optimal to have similar group sizes to maximize power.
2. It is always better to have a **larger total sample size**.
This also means that if you have a fixed number of users that can be exposed to
the treatment (for example, due to legal or budget constraints), the larger the control
group, the better. In other words, if the size of one group is fixed for some
reason, increasing the size of the other group will always improve power.
This is because we want to minimize the uncertainty of the mean for both the
treatment and control groups to accurately estimate the treatment effect.
### Risky treatments
If a treatment is risky, you might want to limit the number of users exposed to
this treatment for risk mitigation purposes. In such cases, the power can be
improved by increasing the size of the control group. Sometimes for risky
treatments at Spotify, the treatment group is fixed to a small size, and the
control group is increased to as large as possible to maximize power. This
requires some fiddling in practice since the allocation of the population and
treatment proportions are both relative.
For example, if the population is 1000 users and you want 30 to be exposed to
the treatment. Then you could have any number up to 970 in the control group. You
could run a 50/50 split on 6% of the population to have 30 in each group, or a
97/3 split on 100% of the population to have 970 in the control group and 30 in
the treatment group.
---
[Image: Optimal Group Sizes for Power]
---
**Quiz: What is the optimal group size allocation for continuous metrics when the total sample size is fixed?**
- A 99/1 split between control and treatment groups
- A 75/25 split between control and treatment groups
- Equal group sizes between control and treatment groups (correct)
- The allocation depends on the baseline proportion
**Quiz: Why is it better to have a larger total sample size in experiments?**
- It minimizes the uncertainty of the mean for both treatment and control groups, improving estimation of the treatment effect (correct)
- It allows for drastically unequal group sizes without losing power
- It eliminates the need for a control group altogether
- It reduces the Minimum Detectable Effect (MDE) for binary metrics
**Quiz: What happens to the required sample size if the relative sizes of the treatment groups are uneven?**
- It increases because uneven group sizes provide less information about the treatment effect (correct)
- It decreases because the control group provides all the necessary information
- It stays the same as long as the total sample size is fixed
- It depends only on the number of users exposed to the treatment
---
## Note for nerds
It's in fact quite straightforward to derive the optimal group sizes for binary
and continuous metrics. If calculus is not your thing, feel free to skip this
section.
### Binary metrics
Let's derive the optimal proportions for binary metrics step by step:
#### Initial setup
Let $N_a$ and $N_b$ be the sample sizes of two treatment groups, and $p_a$ and
$p_b$ be the baseline proportion and the proportion under the hypothetical
treatment effect. Define $\kappa = N_b / N_a$, where $\kappa > 0$. For
simplicity, let $v_j = p_j(1 - p_j)$ for $j \in \{a, b\}$.
#### Step 1: Express total sample size
The minimum required sample size for given type-I and type-II risks is found by solving:
$$\argmin_{\kappa} N = \argmin_{\kappa} \left(\left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 \times (v_a / \kappa + v_b) + \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 \times (v_a + v_b \kappa)\right)$$
This expands to:
$$\argmin_{\kappa} N = \argmin_{\kappa} \left(\left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_a / \kappa + \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_b + \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_a + \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_b \kappa\right)$$
#### Step 2: Take derivative
Taking the derivative with respect to $\kappa$:
$$\frac{\partial}{\partial \kappa} = -\left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_a / \kappa^2 + \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_b$$
#### Step 3: Set to zero and solve
Setting to zero:
$$-\left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_a / \kappa^2 + \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_b = 0$$
$$\left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_a / \kappa^2 = \left(\frac{Z_{\alpha}+Z_{\beta}}{p_a-p_b}\right)^2 v_b$$
$$v_a / \kappa^2 = v_b$$
$$v_a / v_b = \kappa^2$$
$$\kappa = \sqrt{v_a / v_b} = \sqrt{\frac{p_a(1-p_a)}{p_b(1-p_b)}}$$
This implies that for a baseline proportion $p_a$ and a hypothetical treatment group proportion $p_b$, it is optimal to have:
$$N_b = N_a \sqrt{\frac{p_a(1-p_a)}{p_b(1-p_b)}}$$
Clearly, if $p_a \approx p_b$, then $N_a$ is close to $N_b$, which makes the
rule of keeping the groups similar a good general guideline. For the nerds who
paid attention in the previous lesson, this also of course implies that for
binary guardrail metrics, the optimal group sizes are equal.
### Continuous metrics
For continuous metrics, let's derive the optimal group sizes step by step:
#### Initial setup
Let $m_a$ and $m_b$ be the mean of two groups on some continuous metric. Since
the variance doesn't depend on the treatment effect, our optimization
simplifies.
#### Step 1: Express total sample size
We want to minimize:
$$\argmin_{\kappa} N = \argmin_{\kappa} \left(\left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \times \sigma^2 (1 + 1 / \kappa) + \left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \times \sigma^2 (\kappa + 1)\right)$$
This expands to:
$$\argmin_{\kappa} N = \argmin_{\kappa} \left(\left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2 + \left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2 / \kappa + \left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2 \kappa + \left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2\right)$$
#### Step 2: Take derivative
Taking the derivative with respect to $\kappa$:
$$\frac{\partial}{\partial \kappa} = -\left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2 / \kappa^2 + \left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2$$
#### Step 3: Set to zero and solve
Setting to zero:
$$\left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2 / \kappa^2 = \left(\frac{Z_{\alpha}+Z_{\beta}}{m_a-m_b}\right)^2 \sigma^2$$
$$\sigma^2 / \kappa^2 = \sigma^2$$
$$\kappa^2 = \frac{\sigma^2}{\sigma^2}$$
$$\kappa = 1$$
This implies that it is optimal to have equal group sizes ($N_a = N_b$).
### Summary
+ For **binary metrics**, the treatment effect impacts the variance, so the optimal group sizes depend on the baseline proportion and the MDE.
+ For **continuous metrics**, equal group sizes are always optimal.
+ In all cases, a **larger total sample size** will improve power.
---
Source: https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/variance-reduction
# Lesson 3: Variance reduction
> This lesson explains how variance reduction (using regression adjustment)
affects the required sample size calculation. The bottom line is that
variance reduction allows for a smaller sample size to achieve the same power.
---
As you learned in
[Lesson 2](./../sample-size-calculation-i/mean-variance-and-mde)
in the level I course on sample size calculation, the variance of the metric
impacts the required sample size. The larger the natural variation in a metric
across users, the larger samples we require to power the effect (MDE) that we
are interested in. In this lesson, you will learn how variance reduction can
affect the required sample size calculation.
## Variance reduction
Variance reduction has become an umbrella term for any technique that reduces
the variance of the treatment effect estimator as compared to the
difference-in-means estimator.
### Regression adjustment
**Note:** Regression adjustment was popularized for online experiments as 'CUPED'
(Controlled-Experiment using Pre-Experiment Data) by Deng et al. (2013).
Although that paper develops new methods for ratio metrics, and points out the
efficiency of using pre-exposure data of the same metric as the covariate, the
idea of using regression adjustment to reduce variance in randomized
experiments has been around for decades dating at least back to the 1930s.
The idea with regression adjustment is to use covariates that can explain
variation in the metric that is not due to the treatment. This is done by
fitting a regression model that includes the treatment assignment as a predictor
variable and the metric as the outcome variable. The treatment effect is then
estimated as the coefficient of the treatment assignment variable in the
regression model.
The variance reduction factor is simply the proportion of the variance in the
metric that can be explained by the covariates. The variance reduction factor is
always between 0 and 1. The closer it is to 1, the more variance we can explain
and the more we can reduce the variance of the treatment effect estimator. If
the variance of the treatment effect estimator is `X`, and the variance reduction factor is `V`, then
the variance of the variance-reduced treatment effect estimator is `X(1-V)`.
Since the required sample size is a linear function of the variance of the
treatment effect estimator, this means that the required sample size is also
affected linearly by the variance reduction. In other words, if a metric has 40%
variance reduction, then the required sample size is 40% smaller than if we had
no variance reduction. In other words, variance reduction might as well be called
"sample size reduction" for the purpose of experimentation.
**Note:** For efficient experimentation with as
small sample sizes as possible, a metric with slightly higher variance than
another can still be more efficient if it has a higher variance reduction
factor.
---
**Quiz: How does variance reduction affect the required sample size in experiments?**
- It increases the required sample size to achieve the same power
- It allows for a smaller sample size to achieve the same power (correct)
- It has no effect on the required sample size
- It only affects continuous metrics, not binary metrics
**Quiz: What is the variance reduction factor in regression adjustment?**
- The proportion of variance in the metric explained by the treatment effect
- The proportion of variance in the metric explained by the covariates (correct)
- The overall variance of the metric after regression adjustment
- The ratio of natural variance to explained variance in the metric
**Quiz: If a metric has a variance reduction factor of 40%, how does this impact the required sample size?**
- The required sample size is reduced by 40% (correct)
- The required sample size is increased by 40%
- The required sample size is unaffected
- The required sample size doubles
---
## Notes for nerds
The reason why we can "switch" the estimator of the treatment effect is because there are several unbiased estimators of the estimand we are interested in. Estimand is a fancy word for "causal effect of interest", and is used extensively in the economics literature.
Note the wording here, it is the treatment effect estimator and its variance that we are concerned with. This variance is a function of the variance of the metric and the sample size.
Technically, when we say that we "reduce the variance of a metric" what we really do is change the estimator of the treatment effect. Although there are many estimators for the treatment effect in an A/B test besides just the difference in means, 'variance reduction' usually refers to the use of regression adjustment to estimate the treatment effect.
CUPED isn't exactly the same as regression adjustment, because
instead of adding the covariate to a regression together with a treatment dummy
variable, two steps are taken: First the outcome is regressed on the
covariate and then the residuals are regressed on the treatment dummy, or equivalently, the
difference in means are calculated for the residuals. Two great reads on
regression-type adjustments are [Negi and Wooldrige (2020)](https://www.tandfonline.com/doi/abs/10.1080/07474938.2020.1824732) and
[Jin and Ba (2021)](https://arxiv.org/abs/2110.13406).
---
## Advancing your experimentation
Source: https://confidence.spotify.com/bootcamp/advancing-experimentation/introduction
# Advance your experimentation
[Interactive: CourseProgressBar]
In this course, you learn about ways to advance your experimentation practices
and take your learnings from experiments to the next level. This includes
building intuition for and learning how to use the settings in Confidence to
achieve more efficient experiments, better learnings, and more sophisticated
risk management.
**Note:** There are reader exercises at the end of each lesson that helps you check your
understanding. Answer all questions correctly to finish the lesson and track your progress.
This course consists of the following lessons:
- Lesson 1: Guardrail metrics with non-inferiority margins: Learn how to improve the quality of your decisions and the risk management of your experiments by using guardrail metrics with non-inferiority margins.
- Lesson 2: Choose evaluation frequency: Learn about the trade-offs between these settings and optimize your experimentation by choosing the configuration that fits your needs.
- Lesson 3: Metrics' roles in experiments: Learn about how to use the different kinds of metrics in Confidence to optimize your experimentation efficiency.
- Lesson 4: Cumulative holdback evaluations: Learn how to measure the combined impact of all changes shipped during a quarter using holdback groups.
---
Source: https://confidence.spotify.com/bootcamp/advancing-experimentation/choose-evaluation-strategy
# Lesson 2: Choose evaluation strategy: Sequential or non-sequential tests
> What evaluation frequency you choose determines if your experiment uses a sequential or a non-sequential test. Optimize your experimentation by choosing the test that fits your needs.
* Sequential tests let you look at your results and make decisions during the experiment but have lower power than non-sequential tests.
* Non-sequential tests only let you look at the results and make decisions once the experiment has ended but has higher power than sequential tests.
In this lesson you learn how to select evaluation frequency and the various trade-offs to consider when making this choice.
This video gives a 2 minutes and 10 seconds overview of the evaluation frequency topic.
[Video: https://www.youtube.com/watch?v=bMmlPKTt4V8]
## The evaluation frequency determines the statistical test
The evaluation frequency has two options:
- **Continuous**: lets you see results during the experiment
- **Upon conclusion**: lets you see results only when the experiment has ended
### Continuous evaluation uses sequential tests
When you select **continuous** evaluation, Confidence automatically uses a sequential test behind the scenes. Sequential tests ensure that the repeated testing implied by calculating results continuously throughout the experiment does not inflate the risk of finding a false positive result.
There are two types of sequential tests:
- **Group Sequential Tests**: require the experimenter to provide a maximum sample size and must be stopped once this sample size is reached; they have higher power than Always Valid Tests for a given sample size
- **Always Valid Tests**: do not require any pre-specified sample size and can keep letting new users into the sample indefinitely; they have lower power than Group Sequential Tests for a given sample size
**In Confidence:** Confidence automatically chooses the type of sequential test based on whether you provide a maximum sample size or not. If you provide an expected sample size, Confidence uses a Group Sequential Test to maximize power; otherwise it uses an Always Valid Test.
### Upon Conclusion evaluation uses non-sequential tests
When you select **Upon Conclusion**, Confidence automatically uses a fixed horizon test—the classical statistical tests from intro statistics courses.
Fixed horizon tests, such as the z-test, do not allow you to see the results during the experiment, but they have higher power for the given sample size after you stop the experiment and run the analysis as compared to the sequential tests. Although you should run sample size calculations before the experiment, you can view the powered effect at any point during the experiment, which makes it possible to judge when it is time to end it.
## A note on quality tests during the experiment
Regardless of your choice of evaluation frequency, a well-designed experimentation platform uses continuous evaluation (sequential tests) on all quality and deterioration checks during the experiment. This means that you will not miss any experiments that deteriorate or have fatal errors, even if you choose **Upon Conclusion** as your evaluation frequency.
This means the only reason to select continuous evaluation is if you want to make a shipping decision as soon as possible.
**In Confidence:** When you create an A/B test or rollout in Confidence, you can select the [evaluation frequency](../../docs/stats/sequential-tests) in the experiment settings.
## A note on early stopping to ship a variant
Using sequential testing to detect harm and errors to abort failing experiments
early is standard in modern A/B testing. However, stopping early to ship is less
straightforward. This is because it is often easier to prove that something is
too bad to ship than to prove that it is sufficiently good to ship.
A few things to consider are:
- **Time effects.** If there are strong effects of for example weekdays it might be
important to run the experiment over a full week to average these out
- **Novelty effects.** If there is a strong novelty effect, it might be important to
observe users for a longer time to determine the longer time effect of the
change
- **Power.** Under powered experiments tends to overestimate the treatment effect.
If the effect is larger than you expect early in the data collection, you
might want run it longer to ensure your experiment is well powered.
## How to choose
Selecting evaluation frequency and thereby type of statistical test is a
trade-off between various interests. Some general guidelines are:
- The ability to abort failing experiments should not influence the choice of evaluation
frequency. All experiments have sequential checks for deterioration and
quality errors, regardless of the result evaluation frequency.
- If speed of decision making is more important than the quality of the estimates, choose continuous evaluation
- If you are using continuous evaluation, provide a maximum sample size if you can to use Group Sequential Tests that maximizes the power
- If you are going to run the experiment for a fixed number of days anyway
before making the decision to ship, use Upon Conclusion evaluation
For a guide on how to read and act on results depending on which evaluation strategy your experiment uses, see [Lesson 9: Sequential and non-sequential tests](../../bootcamp/interpreting-experiment-results/sequential-vs-non-sequential) in the Interpreting experiment results course.
The recommendation to experimenters at Spotify is to use the upon-conclusion evaluation frequency.
This is because Confidence offers continuous deterioration and quality tests regardless of
the evaluation frequency, there are many reasons to run tests for at least a
fixed time period such as time effects and novelty effects, and upon-conclusion maximizes the chances to detect a true effect (power).
If you are interested in reading more about types of sequential tests and how
they compare, checkout this Spotify engineering blog post about
[choosing a
Sequential Testing
Framework](https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/)
There is also more details about the tests in
[the
documentation](../../docs/stats/sequential-tests).
**Quiz: When should you choose continuous evaluation?**
- If you want to be able to abort as soon as there is evidence of errors
- If speed of reaching a decision is the priority for you. (correct)
- If the most important thing is accuracy in the estimates
**Quiz: What is true for experiments using upon-conclusion evaluation?**
- They risk running for a long time with errors that you do not detect.
- The have less power than continuously evaluated experiments for the same sample size.
- They maximize the power for a given sample size. (correct)
---
Source: https://confidence.spotify.com/bootcamp/advancing-experimentation/guardrail-metrics-and-nims
# Lesson 1: Guardrail metrics with non-inferiority margins
> Guardrail metrics can be tested with inferiority tests or non-inferiority
tests. Using non-inferiority tests improves the quality of the decision to
launch a feature, but require you to specify a non-inferiority margin.
In this lesson you learn how to test guardrail metrics and what the difference between inferiority and non-inferiority tests is.
You can test guardrail metrics in two different ways:
- Use an **inferiority test**. This test evaluates whether there is evidence that the guardrail
metric does **worse** in the treatment group compared to the control group.
- Use a **non-inferiority test**. This test instead evaluates whether there is evidence that the
guardrail metric does **better than a pre-defined threshold** in the treatment group compared to the
control group.
In practice you choose between the two tests by specifying or not specifying a
non-inferiority margin. If you specify a non-inferiority margin, you use a
non-inferiority test. If you don't, you use an inferiority test.
**In Confidence:** Read more about the statistical details of inferiority and non-inferiority tests [in the Confidence documentation](../../docs/stats/stat-tests).
## Inferiority test
The inferiority test seeks evidence that the treatment group does worse than the control group. The
test means that if the metric deteriorates due to the treatment, the test signals that the change causes the metric to
deteriorate and is a not a good launch candidate.
The downside of the inferiority test is that it can never establish evidence that the change *is* safe
to launch. It instead views *lack of evidence for a negative impact as that the change is safe* to launch.
**Example: Checkout flow**
Your guardrail metric number of purchases per visitor changes by -0.2% with a
95% confidence interval of [-0.7%, 0.3%]. There is no evidence that your
change has a negative impact on the number of purchases per visitor. There is
also no evidence that your change doesn't have a negative impact on the
metric.
**If you can't prove that the metric deteriorated, then you conclude that the change is safe to launch.**
## Non-inferiority test
The non-inferiority test seeks evidence that the difference between the treatment and control groups
is not worse than a pre-specified margin, known as the non-inferiority margin. The test means that a
significant result is evidence that the change does better than the margin, and is a good launch
candidate.
**Example: Checkout flow**
Your guardrail metric number of purchases per visitor changes by -0.2% with a
95% confidence interval of [-0.7%, 0.3%]. Your non-inferiority margin is -1%,
meaning that you are willing to accept a 1% decrease. The lower bound of the
interval exceeds this value and you can conclude that there is evidence that
the change does better than your margin.
One way to think about the non-inferiority margin is as a threshold for how much a metric can deteriorate before you consider it to be a negative impact.
**If you can prove that the metric did not deteriorate by more than the non-inferiority margin, then you conclude that the change is safe to launch.**
**Learn about what the non-inferiority margin is and how to set it in 3 minutes and 44 seconds.**
[Video: https://www.youtube.com/watch?v=AIliyHrZa94]
## How Inferiority and non-inferiority tests compare
To illustrate the difference between the two tests, consider the following illustration.
The figure shows a confidence interval for a guardrail metric. The non-inferiority margin is below
the interval's lower bound, which means that there is evidence that the difference between the
treatment and control groups is not worse than the margin. In other words, the change in the metric
is significantly non-inferior, and this is a successful non-inferiority test. If you use the
non-inferiority test, the conclusion is that there is evidence that the change doesn't have a
negative impact on the metric beyond the margin (non-inferiority margin) that you set.
For the same illustration, if you use an inferiority test (disregard the non-inferiority margin),
the conclusion is that there is no evidence for deterioration (as the confidence interval covers
zero), and thus the conclusion is that the change is safe to launch.
### From non-inferiority to inferiority
If you use a non-inferiority test for a guardrail, Confidence still automatically checks if the
metric has deteriorated as part of its monitoring checks.
Consider the following illustration. In this case the metric has not deteriorated beyond the
non-inferiority margin, but it has deteriorated in relation to zero. The overall conclusion is that
it is not safe to launch this feature.
**In Confidence:** In Confidence, even when you use a non-inferiority test for a guardrail, Confidence automatically checks for deterioration as part of its [monitoring checks](../../docs/monitoring#deterioration-checks). Read more about how the various checks feed into [the recommendation to launch](../../docs/analyze-an-experiment#spotlight).
## Practical considerations
### Inferiority or non-inferiority test
Inferiority tests can be a good starting point for guardrail metrics as they don't require the
experimenter to specify a non-inferiority margin. However, the drawbacks are:
- **Non-significant doesn't mean neutral.** You shouldn't make a product decision on a
non-significant result. Lack of significance means you don't have evidence that the metric has
deteriorated. It doesn't mean you have evidence that the metric hasn't deteriorated. With a wide
enough confidence interval, as when, for example, you have a small sample size, everything can be
neutral.
- **A non-inferiority test is a better, but more complicated, approach.** With a non-inferiority
test, you seek evidence that the change is safe to launch. This means you have a specified certainty
that your change doesn't decrease your metric by more than a margin you've decided.
### Set the non-inferiority margin
You need to consider both the business perspective and the statistical perspective when you set the
non-inferiority margin. The business perspective is about what is an acceptable decrease for the
business. The statistical perspective is about what is a reasonable margin given the variability of
the metric and the sample size you can expect.
To decide on the non-inferiority margin, you can consider the following:
- What negative change in the metric is acceptable?
- What change in a given metric would you consider to be so small as to deem it practically
equal?
**Quiz: What does the non-inferiority margin represent?**
- How much we want the guardrail metric to improve.
- How much we want the guardrail metric to deteriorate.
- We want to find evidence that the guardrail metric has not deteriorated more than the non-inferiority margin. (correct)
**Quiz: What is true for a guardrail metric that uses non-inferiority tests?**
- If the metric deteriorates, the inferiority test detects the regression and alerts the experimenter. (correct)
- If the metric deteriorates, we will never know since we use a non-inferiority test.
- If the metric improves, this will lead to rejection of the feature. We want non-inferiority, not superiority.
---
Source: https://confidence.spotify.com/bootcamp/advancing-experimentation/holdbacks
# Lesson 4: Cumulative holdback evaluations
> A cumulative holdback test measures the combined impact of all changes shipped
during a period, such as a quarter. Setting one up requires planning and
coordination to ensure all experiments during the period exclude holdback
users.
## What is a holdback group?
A holdback group is a set of users who are excluded from receiving any product changes during a defined period. This lets you run a test at the end of the period where the holdback group receives all the changes at once, measuring their combined impact.
When you create a holdback, you assign a percentage of users to it.
**In Confidence:** Create a holdback group on the settings page of any surface: [**Surfaces**](../../docs/surfaces) tab > your surface > [**Settings**](../../docs/surface-settings#holdbacks).
## What is a cumulative holdback test?
A cumulative holdback test measures the total impact of multiple changes shipped during a holdback period. After the period ends, you run an A/B test where the holdback group receives all the changes at once. The result reveals the combined effect of everything that shipped.
This is useful when individual A/B tests cannot capture interaction effects between features, or when you want a single measurement of quarterly progress.
## How to set up a holdback
Setting up a holdback requires two decisions before experiments start: how large to make it, and how to create and configure it.
### Determine the size of the holdback group
Choosing the holdback group size involves several trade-offs. A larger group gives more statistical power for the cumulative evaluation, but it reduces the population available for experiments during the holdback period. There is also an opportunity cost: if a shipped feature turns out to be very successful, holdback users miss out on it for the entire holdback period. A large holdback group with a successful quarter of shipping can mean meaningful lost value for those users.
If you're unsure how large to make the holdback, work backwards from the evaluation you plan to run at the end of the period. Set up a placeholder A/B test for that evaluation and use the sample size calculator to find how many users you need to detect the effects you care about. That number tells you the minimum holdback size.
**In Confidence:** In Confidence, create a placeholder A/B test for the holdback evaluation and use the [sample size calculator](../../docs/sample-size-calculator) to determine how many users you need. Use that to set the holdback group size before experiments start.
### Create the holdback group
Create a holdback group in your experimentation tool, giving it a name and a percentage of users.
Some platforms let you make the holdback **required**, so that all experiments automatically exclude holdback users and experimenters cannot opt out. This removes the most common source of holdback corruption: experiments that accidentally or intentionally skip the exclusion.
**In Confidence:** In Confidence, you can make a holdback [required on a surface](../../docs/surface-settings#holdbacks) (global or local). When required, every experiment on that surface automatically excludes holdback users. There is no need to configure each experiment individually, and experimenters cannot accidentally or intentionally skip it.
## Run A/B tests and rollouts during the holdback period
All A/B tests and rollouts during the holdback period must be configured to not overlap with the holdback. Configure this on each experiment when you set it up. If the holdback is required, this happens automatically when a surface is selected.
For rollouts, the reach percentage is calculated relative to the total user population. Since holdback users are excluded, the maximum reachable population is smaller.
**Example: Rollout reach with a 10% holdback**
A 10% holdback group exists. A monitored rollout is configured to not overlap
with the holdback, so it can only reach the 90% of users outside it. The
rollout uses a 90% treatment / 10% control split internally. With 90% of
users available, the maximum treatment reach is 81% of total users, since 90%
× 90% = 81%. The rollout control takes the remaining 9%, and the 10% holdback
group is untouched. The only way to rollout across the holdback's users is to
remove the holdback.
## What changes to include
For a cumulative holdback to be meaningful, all product changes during the holdback period should respect the holdback. This includes:
- A/B tests followed by rollouts
- Direct rollouts to ship a change without a prior A/B test
Minor bug fixes and backend migrations generally don't need to be included, since they aren't made to improve the product. However, any backend change expected to affect the user experience should be shipped through your experimentation platform so holdback users aren't affected.
Don't select which changes to include based on their expected impact. Excluding changes because their impact seems small compromises the purpose of the holdback.
## Measure the impact at the end of the period
After the holdback period is over, run an A/B test targeting only the holdback group. Half of the holdback group receives all the changes that shipped during the period, while the other half acts as control.
**In Confidence:** In Confidence, select your holdback from the **Holdback users** sidebar in the [**Target audience**](../../docs/audience) section of the experiment setup.
## Run holdbacks continuously
The main challenge with back-to-back quarterly holdbacks is creating the new holdback group in time and managing the transition between periods. Create the next quarter's holdback group before the current quarter ends, so any A/B test that might run into the next quarter can be configured against both holdbacks from the start. The two holdback groups should be non-overlapping—each user belongs to at most one of them—so that the groups remain clean and the evaluations don't interfere with each other.
**Example: Transitioning from Q1 to Q2 holdbacks**
**Before Q1 starts**: Create holdback HQ1. Configure all experiments to not
overlap with it.
**Day 50**: A new A/B test is planned that may run into Q2. Create holdback
HQ2. Configure this test to not overlap with both HQ1 and HQ2.
**Day 91**: Q1 ends. Launch the cumulative evaluation A/B test, targeting
only users in HQ1.
**Day 121**: The HQ1 evaluation ends. Remove HQ1, and roll out the Q1
changes to 100%.
Note that the total holdback period includes both the holdback itself and the evaluation test that runs at the end. For example, a quarterly holdback followed by 3 to 4 weeks of evaluation means users in the holdback group are withheld from product changes for a quarter plus those additional weeks.
This video gives a 3 minutes and 57 seconds overview of exclusivity groups and holdbacks.
[Video: https://www.youtube.com/watch?v=8UTi93Yx5vk]
## Cross-surface holdbacks
**In Confidence:** In Confidence, holdbacks can span multiple [surfaces](../../docs/surfaces). A holdback is created on one surface, but experiments on other surfaces can also respect it by selecting both their own surface and the surface where the holdback lives. Read more about [using surfaces to coordinate experiments](../../docs/organize-experiments-with-surfaces). Cross-surface holdbacks require coordination across all teams involved—set up regular syncs to ensure everyone knows which holdbacks are active and when evaluation periods begin and end.
This video gives a 4 minutes and 47 seconds overview of advanced experiment coordination using exclusivity groups and holdbacks across multiple surfaces.
## Risks and common questions
**Holdback corruption.** If an experiment runs without respecting the holdback group, the holdback is contaminated and the cumulative test loses accuracy. Reduce this risk through careful communication, off-platform coordination, and reviewing experiment setups before launch.
**Some changes can't be held back.** Some backend and infrastructure changes can't be withheld from users. When this happens, the cumulative measurement doesn't capture the full impact of all changes. Document which changes couldn't be held back and communicate the caveat when reporting results.
**Which version of a feature to use.** When a feature evolves across multiple holdback periods, use the version shipped at the end of the previous holdback as the baseline for holdback users in the next period. Using the previous quarter's final version as the baseline evaluates the change between quarters rather than each incremental update within a quarter.
**An A/B test runs into the next quarter.** If a test is designed to not overlap with the current quarter's holdback and succeeds, it will be rolled out in the next quarter. Create the next quarter's holdback group early and configure the test to also exclude those users. This ensures that holdback users in both periods are not exposed to the change before the evaluation.
**Quiz: Which product changes during a holdback period should be configured to respect the holdback?**
- Only changes that are directly related to the metrics you plan to evaluate.
- All intended product improvements, regardless of their expected impact. (correct)
- Only changes that are expected to have a large impact on users.
**Quiz: How should you decide on the size of a holdback group?**
- Always use 10% as the standard holdback size.
- Balance the statistical power needed for the evaluation against the opportunity cost of withholding successful features from holdback users and the population available for experiments. (correct)
- Use the same size as the smallest A/B test planned during the holdback period.
---
Source: https://confidence.spotify.com/bootcamp/advancing-experimentation/metric-roles
# Lesson 3: Different types of metrics in experiments
> Required metrics, success metrics, guardrail metrics, and exploratory metrics serve different
purposes and knowing when to use which is important to run efficient
experiments.
The goal of the statistical analysis in an experiment is to provide a solid
foundation for product decisions, covered in detail in
[a previous blog post.](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics/)
This means that the statistical analysis focuses on an overall decision,
informed by all metrics in the experiment. A good experiment uses different
types of metrics to balance the validity of the product decision with learning
as much as possible, safeguarding against regressions, and inspiring new
iterations. Learn how to use different types of metrics to maximize efficiency
and to avoid inflating the sample sizes you need to collect.
Metrics in an experiment can be added in the following ways:
- **Required metrics on surfaces**. Configure metrics that all experiments on specific surfaces must add. The platform verifies that these metrics don't deteriorate.
- **Success and guardrail metrics**. Add the metrics meant to inform the decision when setting up the experiment.
- **Exploratory metrics**. Create exploratory analyses to dig deeper and split by dimensions during or after launching your experiment.
## Use required metrics to align decision scrutiny
Required metrics are metrics that all experiments that run on a surface check for regressions. These metrics show up on the experiment design page automatically when you select the surface.
Importantly, required metrics are only checked for deterioration—Confidence recommends aborting the experiment if a required metric moves significantly in the unintended direction. These metrics have a negligible effect on the required sample size of the experiment. See
[this blog post](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics/)
for more details.
Required metrics increase the autonomy of teams. A single team focuses on
optimizing the user experience they own, using metrics related directly to that
experience. If an experiment has an unintended effect on
business-critical metrics, Confidence alerts the team.
**In Confidence:** In Confidence, [required metrics](../../docs/surfaces#required-metrics) are configured on surfaces and show up automatically in the experiment design page.
## Use success and guardrail metrics to construct the basis for your decision
### Success metrics
For success metrics, the recommendation is to ship if at least one success
metric has significantly improved. This means that for each success metric you
add, you have one more chance for a false positive result. The platform
corrects for the number of success metrics to control the overall false positive
rate, leading to a higher required sample size.
### Guardrail metrics
For guardrail metrics, you can choose to use them in one of two ways:
- With non-inferiority margins
- Without non-inferiority margins
Learn more about the two ways to analyze guardrail metrics in
[the lesson on the topic](./guardrail-metrics-and-nims).
If you use non-inferiority margins, which is the most rigorous way of using
guardrail metrics, all guardrail metrics must be simultaneously powered. You should only
ship if the treatment is significantly non-inferior to control
for all guardrail metrics. Each guardrail metric adds a chance of not finding a
significant non-inferior result, which means that the power per metric is corrected upwards, leading to a higher required sample size.
If you don't use non-inferiority margins for guardrail metrics, they are only
checked for significant regressions. Interpreting the lack of evidence for
deterioration as a signal for shipping the variant is a less rigorous way
to work with guardrail metrics. Guardrail metrics without
non-inferiority margins don't affect the required sample size. This means that
if the sample size requirements are too large when all guardrails have
non-inferiority margins, you can trade off rigor for a smaller sample size by
changing some guardrail metrics to not use non-inferiority margins.
## Use exploratory metrics to learn more
You can add any metrics to an exploratory analysis of the experiment. You can
also slice and dice these results on dimensions to see the results in
different subgroups. Use explorations to dig deeper into the
results after you make a decision from the experiment. The
exploration is separate from the analysis on the result page, and does
not affect the required sample size of the experiment.
**In Confidence:** In Confidence, use the [Explore tab](../../docs/exploration) to add metrics and slice results by dimensions.
## Recommendations
Consider the following recommendation when you configure metrics for your experiments:
- **Add as required metrics** the metrics that should never deteriorate, but that you rarely use as decision metrics in experiments.
- Add them to the global surface if they apply business wide, or on a local surface.
- Required metrics don't inflate the required sample size, but adding more decreases the chances of finding a deterioration.
- **Add as success metrics** the metrics that quantify the success of a treatment.
- These should generally be few.
- The more success metrics you add, the larger the sample size you need to collect is.
- **Add as guardrail metrics with non-inferiority margins** the metrics that your success metrics are likely to cannibalize.
- The margin lets you quantify the trade-off between a guardrail metric deteriorating and a success metric improving.
- The more guardrail metrics with non-inferiority margins that you add, the larger the sample size you need to collect is.
- **Add as guardrail metrics without non-inferiority margins** the metrics that you don't want to see deteriorate, but that you have a less strict trade-off for.
- Guardrail metrics without non-inferiority margins don't inflate the sample size you need to collect, but detecting true deterioration becomes harder the more you add.
- **Add as exploratory metrics** the metrics that you don't base your decision on, but that help you learn and understand the results of the experiment.
- Exploratory metrics don't influence the sample size you need to collect, but adding more makes it harder to find effects in your explorations.
**Quiz: What is a true statement about required metrics?**
- They improve the power of the experiment.
- They don't affect the required sample size of the experiment. (correct)
- They are selected by each experimenter
**Quiz: When should a guardrail metric be used without non-inferiority margins?**
- When the metric is highly sensitive and critical to our business.
- When you want to check for regressions but you don't have any strict trade-offs to make between this metric and a success metric. (correct)
- When you use sequential testing.
---
## Experimentation culture
Source: https://confidence.spotify.com/bootcamp/experimentation-culture/introduction
# Build and maintain an experimentation-driven culture
In this course, you will learn strategies to build and maintain an experimentation-driven culture in your organization. This includes onboarding new colleagues, empowering champions, and organizing forums and drives to sustain a robust experimentation culture across teams.
**Note:** There are reader exercises at the end of each lesson to check your understanding. Answer all questions correctly to finish the lesson and track your progress.
This course consists of the following lessons:
- Lesson 1: Onboard new colleagues to experimentation: Learn how to incorporate experimentation into your onboarding programs to build a strong foundation of experimentation know-how across all roles.
- Lesson 2: Empower experimentation champions: Discover how to identify and empower experimentation champions to scale onboarding and drive experimentation practices across your organization.
- Lesson 3: Sustain an experimentation culture: Learn how to use forums and experimentation drives to energize and sustain an experimentation culture throughout your organization.
---
Source: https://confidence.spotify.com/bootcamp/experimentation-culture/champions
# Lesson 2: Empower experimentation champions
> Champions are key individuals who amplify and spread experimentation knowledge across teams. Empowering them can help scale experimentation practices efficiently.
In this lesson, you will learn how to identify and empower champions to drive experimentation culture in your organization. Champions are individuals who are either knowledgeable or enthusiastic about experimentation. Tapping into this interest can help you scale onboarding and practices across teams.
## Steps to empower champions
1. **Identify potential champions** who show interest or expertise in experimentation.
2. **Educate and mentor them** with relevant resources:
- Quickstart guides for your experimentation platform
- Foundational experimentation courses
- Videos on advanced experimentation topics
3. **Support champions** with tailored material based on their role. For instance:
- Engineers: Focus on feature flagging and rollouts.
- Data scientists: Emphasize metrics and analysis.
4. **Facilitate recurring meetings** where champions share their experiences, challenges, and learnings.
5. **Recognize and reward champions** by ensuring their efforts are acknowledged, such as using the role to support promotions.
**In Confidence:** Recommended resources for champions in Confidence:
- [A/B test quickstart](../../docs/abtests/quickstart): Learn how to run and interpret A/B tests.
- [Rollout quickstart](../../docs/rollouts/quickstart): Use feature flags to ship features safely.
- [Introduction to experimentation course](../intro-course/introduction): Gain foundational knowledge.
- [Videos](../videos): Explore short videos on advanced experimentation topics.
Empowered champions become super-experimenters who inspire others and drive cultural change.
**Quiz: What is one key benefit of empowering experimentation champions?**
- It reduces the workload of experimentation teams by scaling onboarding from within. (correct)
- It eliminates the need for onboarding programs.
- It discourages teams from running experiments.
---
Source: https://confidence.spotify.com/bootcamp/experimentation-culture/onboarding
# Lesson 1: Onboard new colleagues to experimentation
> Onboarding new colleagues to experimentation as part of their general onboarding is a powerful way to gradually build a strong foundation of experimentation expertise across the organization.
In this lesson, you will learn how to incorporate experimentation onboarding into the onboarding programs of new colleagues. This ensures that all roles—engineers, product managers, and data scientists—are equipped with relevant experimentation knowledge and skills from day one.
At Spotify, we include experimentation onboarding by providing role-specific quickstart guides and courses. Here are some examples:
## Engineers and developers
Engineers need to understand how to connect their code to the experimentation platform using feature flags, and how to ship safely with rollouts.
**In Confidence:** - [Feature flag quickstart](../../docs/flags/quickstart): Hook up and control code with a feature flag.
- [Rollout quickstart](../../docs/rollouts/quickstart): Ship new features safely with monitoring.
- [A/B test quickstart](../../docs/abtests/quickstart): End-to-end guide to running and interpreting A/B tests.
## Product managers
Product managers benefit from understanding the A/B testing process end-to-end and the key concepts behind experimentation.
**In Confidence:** - [A/B test quickstart](../../docs/abtests/quickstart): Run and interpret A/B tests end-to-end.
- [Introduction to experimentation course](../intro-course/introduction)
- [Scientific product development course](../scientific-product-development/introduction)
## Data scientists
Data scientists need to understand how to create and configure metrics, and how the statistical analysis works.
**In Confidence:** - [Metrics quickstart](../../docs/metrics/quickstart): Create metrics from scratch.
- [A/B test quickstart](../../docs/abtests/quickstart): End-to-end A/B testing experience.
- [Introduction to experimentation course](../intro-course/introduction)
By tailoring onboarding material for each role, we can ensure that every new joiner is equipped to contribute to our experimentation culture.
**Quiz: Why is it important to include experimentation onboarding in the general onboarding of new colleagues?**
- It replaces the need for technical training.
- It ensures that everyone learns experimentation practices gradually and sustainably. (correct)
- It allows teams to avoid experimentation altogether.
---
Source: https://confidence.spotify.com/bootcamp/experimentation-culture/sustaining-the-culture
# Lesson 3: Sustain an experimentation culture
> Forums and experimentation drives are effective ways to sustain long-term engagement and energy for experimentation across an organization.
In this lesson, you will learn how to use forums and experimentation drives to
keep the experimentation culture thriving. These practices provide a platform
for sharing ideas, solving challenges, and driving collaboration.
## Experimentation forums
Regular forums are a great way to inject energy into the experimentation culture. Forums can include:
- Discussions on current or planned experiments.
- Sharing metrics and results interpretation strategies.
- Addressing challenges like sample size and hypothesis testing.
## Experimentation drives
Dedicated time slots, such as hack weeks, help teams focus on experimentation. Example activities include:
- Creating feature flags and metrics.
- Running A/A tests to validate your setup end-to-end.
**In Confidence:** For Confidence users, running A/A tests using the [A/B test quickstart](../../docs/abtests/quickstart) is a great hands-on exercise during experimentation drives.
By combining forums for inspiration with hands-on experimentation drives, you can create a self-sustaining culture of experimentation.
**Quiz: What is the purpose of experimentation drives?**
- To distract teams from their regular work.
- To provide focused time for teams to get started with experimentation. (correct)
- To increase the number of experiments without any planning.
---
## Intro to feature flags
Source: https://confidence.spotify.com/bootcamp/intro-to-feature-flags/introduction
# Introduction to feature flags
[Interactive: CourseProgressBar]
In this course, you will learn about feature flags and how to use them effectively in your applications.
**Note:** There are reader exercises at the end of each lesson that help you check your
understanding. Answer all questions correctly to finish the lesson and track your progress.
This course consists of the following lessons:
- Lesson 1: What is a feature flag: In this lesson, you learn about what a feature flag is and what you can use them for.
- Lesson 2: Life cycle: In this lesson, you learn about the life cycle of a feature flag. This includes when to create, update, and delete feature flags.
- Lesson 3: Clients: In this lesson, you learn about what a client is and why it is often helpful to use different clients for different parts of your application.
- Lesson 4: Evaluation context and targeting: In this lesson, you learn about the evaluation context and targeting of feature flags, so that you can control which users see which flag variant based on their context.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-feature-flags/clients
# Lesson 3: Work with Clients
> Learn what clients are and how you can use them to structure your feature flag integrations.
## What is a client?
A feature flag is not useful for anyone unless it's consumed. The consumer of
this feature flag is not the end user of your product but rather a part of your
application. We call this consumer a client. Most feature flagging systems
support clients of both backend and frontend types, and most provide SDKs to easily integrate.
**In Confidence:** In Confidence, every SDK or API integration can be its own client, letting you control which flags are accessible from which part of your application. You add a Confidence client to your code to handle communication with the resolve API.
## Best practices
There is no benefit to having multiple systems share a single client. Instead, treat every integration as its own client. This allows for a flexible and future proof approach where frontend
clients (using the static paradigm) are not overwhelmed with all the flag
data that is only relevant to backend clients.
In other words, by only letting one app access the flags that are relevant to
it, you can reduce the overhead of having many flags in your systems.
### Example: Manage latency with multiple clients
**In Confidence:** Consider this example: a single Confidence client is shared across 20 backend services and 3 frontend apps, covering 5 flags. Frontend clients evaluate all flags at startup — say 30 ms. Backend services evaluate one flag at a time. Now the backend team creates 100 new flags, all backend-only. Frontend clients still fetch data for all of them, inflating their startup latency for flags they'll never use. Separating clients per integration avoids this entirely — Confidence clients for frontend and backend stay scoped to their own flags.
## Client granularity and security
It's good to have one client per app (backend service or frontend application). With this granularity, you can control the access of individual modules. For instance, you can revoke access to one backend service without stopping all of the others. A single client per app also helps you control what information is exposed to the user. Even if you don't expose all flags directly to the user, any information that reaches a frontend app can be considered exposed. For instance, if you have a flag intended for a backend service called 'enable super secret new feature X' and you use the same client for your frontend, that flag is still fetched and could be introspected by someone snooping around with a browser debugger.
**In Confidence:** In Confidence, each client authenticates separately with Confidence, giving you granular control to revoke access to individual modules without affecting others.
**Note:** It's bad to have more than one client per app. Each client acts as its own
universe. Multiple clients lead to multiple network requests to fetch flags
and keep multiple copies of internal state such as context describing the
module.
**Quiz: What is a client in the context of feature flags?**
- The end-user of an application.
- A specific instance of a web browser.
- A part of your application that consumes feature flags. (correct)
**Quiz: Why is it generally recommended to have one client per module in Confidence?**
- To simplify the codebase and reduce the number of API keys.
- To improve performance by loading all flags for all modules at once.
- To enhance security and control access to flags on a granular level. (correct)
---
Source: https://confidence.spotify.com/bootcamp/intro-to-feature-flags/evaluation-context-and-targeting
# Lesson 4: Evaluation Context and Targeting
> You can give users flag values conditioned on their context. By sending in context with the flag resolve call, the feature flag resolver can use this context and have conditional rules. For example, only users in the US gets a certain experience.
## What is the evaluation context?
Context is data relevant to your users that you are
interested in targeting your experiments on. It can be as generic as their
country or as granular as the amount of times they refresh the landing page. You
have the power to collect as much or as little data you want - but what is the
suggested approach?
The following video provides a quick overview of how the evaluation context is related to targeting.
[Video: https://www.youtube.com/watch?v=rU82f1nRoGo]
The current evaluation context is an integral component in the system and is part of
many SDK calls - so be aware that including too much data in a context could
potentially lead to increased latency. Work with generic structures that
can be reused for targeting across different experiments to minimize the size of
your context.
## Automatic data collection
Some SDKs allow you to automatically collect certain device data:
- Device manufacturer and model
- Application version
- Custom data that you configure to be automatically collected in your application
**In Confidence:** Confidence SDKs support automatic collection of device data (manufacturer, model, application version, and custom data you configure) out of the box.
## Targeting with context
When you're done setting up your context, you're ready to use it for targeting.
In your A/B test or rollout, you can find the relevant data for targeting, such as
country, visitor_id, etc. Keep in mind that to mix and match contexts from
different systems (mobile, web, backend), you need to ensure that the
context is available for all parts of your application where the flag in your
experiment is resolved.
## Frontend vs. backend context
Frontend apps often have a single user that uses the application. For this reason, a 'static context' paradigm is often applied to frontend SDKs. This means that the contextual data collected in the SDK is static and will stay in memory throughout the application lifecycle until actively being changed or removed through the SDK API.
Backend apps/services on the other hand can serve thousands of different end users during the same second, so in this case, a 'dynamic context' paradigm fits better. The dynamic context paradigm favors volatile context values that will only be used for that split second in which your backend service caters to the end user's request.
**In Confidence:** In the Confidence SDKs, frontend clients use a static context paradigm and backend SDKs use a dynamic context paradigm. Backend SDKs also have support for longer-living context data that is contextual to the actual backend application (for example, in which geographic region it exists).
## Context updates and flag re-evaluation
Frontend clients automatically fetch new flag values when a context value is
added, removed or changed. This is because flag values may depend on that
specific context value in its rules setup. For example, targeting criteria
for an experiment may include user information which is only available after the
user has been logged in. The act of logging in would then enable the application to
add user information to the context and the SDK will then fetch new flag values.
Certain care should be taken when working in frontend environments and adding values that
are volatile (likely to change very often). Since every context change triggers a fetch, we
recommend _against_ having volatile values in the context for performance reasons.
An extreme example of this would be to continuously add the mouse location to the context.
## Context and data integrity
The context is data maintained by the client, and it should at all
times reflect truths about the user and/or its device. Which country is the user
registered in, but also which country is the user's device currently accessing
your app from. Which locale has the user selected.
**Quiz: What is the purpose of evaluation context in feature flagging?**
- To store sensitive user data.
- To provide data that can be used to target feature flags to specific users or groups. (correct)
- To track the performance of individual feature flags.
**Quiz: Why is it important to be mindful of the amount of data stored in the context?**
- Large context objects can lead to increased latency. (correct)
- Context data is stored permanently on the users device.
- Storing too much data in the context replicates it to all active SDK instances, increasing network traffic.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-feature-flags/life-cycle
# Lesson 2: Life cycle
> This lesson covers the lifecycle of feature flags, from their creation to
their eventual removal or archival. We'll explore how feature flags evolve
alongside feature development, the differences in managing flags across backend,
web, and mobile environments, and best practices for cleanup and reuse.
## Feature flag evolution
During feature development, it's common to start with a single flag that controls
all configuration for that feature. This flag can hide the feature until it's
ready for release. Using the feature flag, you can remotely toggle the feature
for yourself and selected colleagues during development. As development progresses,
the flag evolves with the feature, and different properties may be added or removed.
By the end, the flag might contain just a simple boolean 'enabled' property, or it
might retain certain aspects that you want users to control through experimentation.
## Example life cycle
To illustrate the life cycle of a feature flag, we will go step by step, from creation
to removal, both in your experimentation platform and your app code.
[Image: Create a flag illustrated]
1. Create a flag
- Create the flag in your experimentation platform
- Add the flag to the app code
- Test it with your team
[Image: Parametrize the user experience illustrated]
2. Parameterize the user experience
- Add properties to the flag and connect them to the feature
- Set default values matching the current experience
- Run A/B tests to optimize property values
- Gradually roll out the winning variant
[Image: Update default values illustrated]
3. Update default values
- Update the app code with the new default values
- Release a new app version
- Wait for user adoption
[Image: Remove a flag illustrated]
4. Remove the flag
- Replace feature flag code with hard-coded values
- Remove the flag from the experimentation platform
**In Confidence:** In Confidence, create the flag in the Confidence app (step 1) and remove it from the Confidence app when cleaning up (step 4).
## Post-experimentation: Flag retention
After completing an experiment and rolling out the winning variant, you have three options:
- Clean up the flag
- Retain the 'enabled' property as a safety switch for quick feature disabling
- Retain the full flag to enable future iteration
Consider these factors when deciding how long to keep a flag:
### Code complexity and bundle size
Feature flags create conditional branches in your code ('if enabled'). Keeping
these branches increases code complexity and bundle size.
**Recommendation:** While feature flags offer great flexibility, maintaining too many can lead to
unsustainable complexity and increased bundle size, as we've learned at Spotify.
### User experience after flag removal
For backend services, you control the code version in production. However, with
mobile apps, you need to consider how removing a flag affects users on older
app versions. You might want to maintain the flag until you achieve high adoption
of the new app version, which can take several months for some apps.
**Quiz: What should you do with a feature flag after an experiment is complete and the winning variant is rolled out?**
- Always delete the flag immediately.
- Keep the flag indefinitely to avoid code changes.
- Consider cleaning up the flag or repurposing it if necessary. (correct)
**Quiz: Why is it important to consider mobile app updates when deciding whether to remove a feature flag?**
- Mobile app updates are irrelevant to feature flag management.
- Older app versions may not be able to handle removed flags, potentially causing issues. (correct)
- Feature flags improve the performance of mobile app updates.
---
Source: https://confidence.spotify.com/bootcamp/intro-to-feature-flags/what-is-a-feature-flag
# Lesson 1: What are feature flags?
> A feature flag is a configuration tool that gives the application owner the
ability to remotely control different ways in which a specific part of the
application should behave-without having to change or deploy new code. A feature flag configures each running application according to certain
criteria, commonly referred to as 'targeting criteria'. Examples of targeting
criteria are: country where the application is running from, version of the
application, etc.
The following video provides a quick overview of what feature flags are and how they can be used:
[Video: https://www.youtube.com/watch?v=E_-U0ryfDPI]
In the common scenario, a feature flag is functionality you add to your code
that lets you control the behavior of some part of your application remotely,
without changing the code. Your application asks a feature flag resolver for
what behavior a certain user should have given a certain context. This is
called "resolving" the feature flag.
Feature flags get much more powerful when paired with an experimentation system
and/or a user-behavior instrumentation system. In this case, you can use the
feature flag to run randomized experiments that let you evaluate the impact of
changing from one behavior to another. This is often referred to as A/B testing.
In this case, different users are given different 'treatments' by receiving
different flag values.
## Use cases for feature flags
Feature flags can be used to:
- Toggle features on and off for certain users.
- Toggle features on for the product development team to QA new features.
- Serve different experiences to different users, and evaluate the most successful experience according to the underlying experimentation system.
- Try out a new design of your UI elements.
- Gradually roll out a new feature and control who gets the feature.
- Adhere to policies that can change over time, for example what people in a specific country can/cannot see.
## Things to be mindful about
When using feature flags that are controlled by an external system there are a
few things to be mindful about. Thinking about these things early in your
feature flag journey can save you a lot of time and headaches later on.
### Default values
It's always important to consider the default scenario so your app continues to
function even though you cannot resolve a specific flag, or if the resolve
returns no value for the flag.
If a feature flag changes/updates in such a way that your client cannot deal
with it you need fallback solutions for that. A typical case for when this can
happen is when you have a mobile application where you cannot control when that
app is updated on the client side, or if the network call to the feature flag
resolver fails or times out. Most feature flag providers have default values defined in the code that help handle these cases.
**Note:** Falling back in default values is not a problem, rather it's a good practice, you just need to be aware of it.
If you are adding a new feature flag, you set the default value to the current hard-coded behavior that the static code implied.
This way, you can merge the feature flag into your codebase without worrying about changing the behavior of the app.
### Flicker
Flicker is a phenomenon where the value of a feature flag is changed rapidly in
a way that is noticeable to the user and gives a flickering experience. For
example, a button might appear and disappear rapidly.
Technically, what often happens is that the code in the app is changed because the
feature flag controlling the code is updated in front of the eyes of the user.
This can happen if a feature flag is not resolved quickly enough (high latency)
or if the feature flag is implemented in the code in a way that doesn't suit the
use case.
**Note:** When the context of the user changes, it might be expected that the feature flag
value changes. Whether this is a problem depends on the use case. See [Lesson 4](./evaluation-context-and-targeting) for more information.
There are many ways that flickering can be avoided. You should ensure that the feature flag provider has latency that is acceptable to you. Beside the latency aspect, you need to ensure that the implementation of the feature flag in the code is correct, and follows the recommended patterns.
**In Confidence:** Confidence provides feature flag resolving in three ways to help minimize latency:
- Online resolving: the standard way, making a network call to the Confidence feature flag resolver
- Sidecar resolver: set up the Confidence resolver as a sidecar in Kubernetes, reducing latency to that of a local call
- Edge resolving: use an edge compute provider like Cloudflare or Fastly to resolve the feature flag on the edge, by running the Confidence resolver in an edge worker
## Complexity levels
A feature flag can be increasingly complex. From a simple boolean on/off flag,
to a 'Super Flags' that contain a lot of configuration information in a
JSON-type object that can be nested. There is no right or wrong answer here and
it depends very much on your situation and needs.
If you are interested in optimizing a certain user experience with A/B testing,
for example the title string of a page, you could use a flag with properties
like `title`, `color`, and `font-size`. Where `title` and `color` are of type
string, and `font-size` is of type integer. With this setup, you can compose
different experiences by changing the title, color, and font-size of the page,
without having to change the code. This enables non-engineers to iterate on an
experience without having to rely on the engineering team.
**Note:** Connecting various aspects of a user experience to a feature flag with several properties is called 'parametrizing' the user experience. You add parameters to certain aspects of the user experience, and then control these parameters remotely via the feature flag.
## Fetch and resolve flags
When your application asks the feature flag resolver for the value of a flag,
there are in fact two steps. The first is fetching the flag which means getting
the correct flag value for a given user in a given context from the API. The
second step is resolving the flag which means applying or using the flag value
in the app.
The separation between fetching and resolving is important because it allows you
to control when you want to get the flag value and when you want to apply it.
Common patterns are to fetch the flags in the background, during the session, or
at startup with a loading screen.
In order to make a decision whether you want to fetch your flags in the
background, in the session or in a loading screen, think about what types of
experiment you want to conduct. If your experiments are targeted on UI changes,
it's advised to fetch and resolve as soon as possible and in bulk - this way,
you can guarantee no flickering and all in all good user experience. If you're
experimenting with platform, backend, etc - you can get away with showing your
app without loading screen and fetching the flags in the background thread
without compromising user experience.
**Quiz: What is a feature flag?**
- A way to hide code.
- A configuration entity that controls application behavior. (correct)
- A type of database.
**Quiz: What is NOT a common use case for feature flags?**
- Running A/B tests.
- Controlling access to premium features.
- Storing user data. (correct)
---
## Videos
Source: https://confidence.spotify.com/bootcamp/videos
# Videos
---
## Frequently Asked Questions
Source: https://confidence.spotify.com/bootcamp
**Who is the Confidence Bootcamp for?**
The bootcamp is designed for anyone who wants to improve their experimentation skills. Courses are tailored for data scientists, analysts, engineers, product managers, and leaders — whether you are running your first A/B test or scaling an experimentation program across your organization.
**Is the bootcamp free?**
Yes, the Confidence Bootcamp is completely free. All 11 courses, 90+ lessons, and resources are available at no cost. You can start learning immediately without creating an account, though signing in lets you track your progress across devices.
**What will I learn?**
The bootcamp covers the full experimentation lifecycle: A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture. It includes 11 courses with over 90 lessons built by the Confidence team at Spotify.
**How long does the bootcamp take to complete?**
The full bootcamp takes approximately 20 hours to complete across all 11 courses. Individual courses range from 30 minutes to 3 hours. You can learn at your own pace and pick the courses most relevant to your role.
**Do I need prior experience with A/B testing or statistics?**
No prior experience is required. The bootcamp starts with foundational courses like Intro to Experimentation and progressively covers more advanced topics like sequential testing and variance reduction. Each course clearly indicates which roles it is designed for.
**Who created the Confidence Bootcamp?**
The Confidence Bootcamp was created by the Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify. The content reflects real-world experimentation practices used at one of the world's largest digital products.
---