Welcome to Interpreting experiment results
Interpreting experiment results is an asynchronous, self-paced course that teaches you how to read and understand the results page in Confidence. By the end of this course, you will be able to look at any experiment results page and know exactly what every number, label, and recommendation means, and what to do with that information.
The course is designed to be accessible regardless of your background or role. You do not need a statistics degree to follow along. Where the details matter, this course explains them in plain language and flags where you can dig deeper if you want to.
Some things may look different from other tools you have used. Where Confidence does things its own way, the approach is grounded in years of iteration and original research.
There are quiz questions throughout the course to help you check your understanding of the material. Complete each lesson's questions to track your progress.
Before you begin
This course works best if you have run at least one experiment in Confidence, or have followed the A/B test quickstart. Having a concrete experiment in mind as you go through the lessons will help the concepts click.
Lessons
This course consists of the following lessons:
Lesson 1: The anatomy of the results page
Get oriented on the three sections of the results page and understand the basic logic connecting them.
Lesson 2: The Spotlight
Understand the overall recommendation (Ship, Continue, End, or Abort) and what drives each one.
Lesson 3: Means and relative effects
Understand what the control variant and treatment variant means represent, and why effects are shown as relative percentages.
Lesson 4: Confidence intervals and precision
Learn what confidence intervals are, how to read them, and why their width tells you how precisely the effect has been measured.
Lesson 5: Significance for success metrics
Understand what 'significant' and 'not significant' mean for success metrics, and how the CI position determines the status.
Lesson 6: Guardrail metrics and NIMs
Learn how guardrail metric status labels work, what a non-inferiority margin is, and why it gives stronger evidence of safety.
Lesson 7: Health checks and the SRM
Learn how Confidence verifies that your experiment is trustworthy, and what to do when a health check fails.
Lesson 8: Variance reduction
Understand why the means shown in results may differ slightly from raw averages, and how to interpret them correctly.
Lesson 9: Sequential and non-sequential tests
Learn when you can trust the results you see, and what your choice of evaluation strategy means for your experiment.
Lesson 10: Exploratory analysis
Learn how to use explorations to learn more from your experiment without drawing false conclusions.
Lesson 11: The winner's curse
Learn about the winner's curse, why significant results from underpowered experiments tend to overestimate the true effect, and how to use confidence interval precision as a practical safeguard.