Lesson 5: Experiment setup

After the planning stage, you need to translate the details of your plan into an actual experiment configuration. At this stage you create an experiment and configure it according to your plan, which includes:

• Configure the test variants: What part of the user experience will the test change for the users?
• Select your target audience: For which users will the experience change?
• Select the metrics you want to track: What are the key success metrics you want to measure? What are important guardrail metrics?

Configure the test variants

To set up an experiment, you need to define the test variants and connect the variants to the product code. Feature flags are the standard mechanism for connecting experiment variants to your product.

Watch this video to get a quick understanding of how feature flags let Confidence control parts of your product remotely.

Select your target audience

When setting up an experiment, you need to define the target audience for the experiment. This is the group of users that will be included in the experiment. You can define the target audience based on different user attributes, such as country, platform, or user age. What you can target on is determined by what information is passed in when you resolve the feature flag.

Watch this video to get a quick understanding of how targeting and feature flag resolving are related.

Select metrics

Refer to Lesson 3 and Lesson 4 for details on the types of metrics and how to select them.

Sample size and design

One important aspect of the experiment setup is the required sample size. The required sample size is the number of users that you need in your experiment to reach a level of precision in your results that lets you answer the question you have set out to answer. You can control the sample size by setting the allocation, the proportion of the target population that your experiment will include.

Learn more about sample size calculations in Lesson 8.