Course wrap up

Congratulations! You have finished Interpreting experiment results!

You can now open any experiment results page in Confidence and know exactly what you are looking at. To recap what you have covered:

• The results page has three sections: Spotlight, Health checks, and Metrics, each answering a different question.
• The control variant and treatment variant means are averages of what actually happened for real users, and effects are always shown as relative % changes to make them comparable across metrics.
• A confidence interval tells you both where the effect likely is and how precisely you have measured it. A wide CI means you need more data, not that there is no effect.
• Status labels differ between success and guardrail metrics because they answer different questions: "did it improve?" versus "did it break anything?"
• The SRM check is the most critical health check. If it fails, no metric result can be trusted.
• Variance reduction makes estimates more precise by using pre-experiment behavior to remove noise. The numbers look slightly adjusted, but you interpret them the same way.
• Your choice of evaluation strategy determines when results are valid to act on. Deterioration checks always run sequentially regardless of that choice.
• The Spotlight synthesizes everything (health, success metrics, and guardrail metrics) into one recommendation per treatment variant.
• Explorations are for learning and hypothesis generation, not for deciding whether an experiment succeeded.

What to explore next

If you want to go deeper on the statistical foundations behind what you learned here, the A primer on hypothesis testing course covers the mechanics of how hypothesis tests work and where p-values and significance thresholds come from.

To learn about more advanced experiment configurations, including guardrail metrics with non-inferiority margins and how to choose between sequential and non-sequential tests, check out Advance your experimentation.

Go back to my learning page to keep learning!