Lesson 10: Run Your Experiment

After you launch your experiment, you should monitor that everything works fine. A good experimentation platform supports you in this stage with automatic monitoring checks. In this section of the course you learn what those checks are, and how to monitor that your experiment delivers your new experience.

Directly after launch

When you launch the A/B test, exposure calculations run at frequent intervals initially. This makes it possible to directly see that the experiment works as intended.

In Confidence

Confidence runs automatic monitoring checks throughout the experiment. Hover the Live status on the right sidebar to see their current status.

If you use Confidence's flags, you can also see the real-time resolve count alongside the exposure schedule.

At any time while the experiment is live

As soon as the first exposure and metric calculation finish, you can check the status of monitoring checks for your experiment.

If you configured your experiment to deliver results continuously, the full results page is available while the experiment is live with results for all metrics that have at least some data.

When you end the experiment

If you configured your experiment to deliver results at the end of the experiment, the results page only shows the results for all metrics after you end it.

You can also use exploratory analysis to further drill down the results of metrics by splitting the results by dimensions, and add new metrics that weren't part of the initial experiment setup. You can, for example, split the results by market to see how results differ between different markets, or add a metric that measures something similar to an existing metric to make sure that an unexpected result is present also in the new metric.

You can use exploratory analysis during the experiment, but to prevent p-hacking and cherry picking of results, you should wait until the experiment has ended before exploring additional metrics.