Skip to main content
By the end of this guide, you have a live A/A test on 5% of your traffic and know how to read its results.
Do it with AI. Ask the Confidence Agent in the app or in Slack:
The agent creates the A/B test. It launches it only after you confirm. Review the configuration in the app before you launch.

​
Evidence For Your Ideas

An A/B test is an experiment that lets you compare user reactions to different variants of your product. You get trustworthy evidence for or against new variants so that you can learn from your users and iterate quickly. Some sections link to associated videos and documentation. Make use of these resources as they contain important information that the guide doesn’t cover. This guide helps you set up a test that doesn’t change anything in your code. Such a test lets you learn how to set up and run an A/B test in Confidence, and goes by the name of A/A test. It can either use the same variant for treatment and control, or use different variants that don’t change the user experience. They help you practice setting up and running A/B tests before you start testing real changes, providing an opportunity to validate and practice your full flow.

​
Before You Begin

This guide requires a feature flag and at least one metric. The guide uses the flag tutorial-feature with control and treatment variants, and the metric Page views per visitor for the Visitor entity. If yours have different names, use your own flags, variants, metrics, and entities wherever the guide refers to these.
The assistant uses the Flags MCP tools createFlag, addFlagVariant, listEntities, and listMetrics. To create the metric with AI, see Create a metric.

​
Create an A/B Test

MCP can’t create A/B tests. Ask the Confidence Agent in the app or in Slack:
The agent can also set the treatments, audience, metrics, and hypothesis from the next sections in the same request. See the prompt in the Do it with AI box at the top of this page.

​
Set the Name, Assignment Table, Entity, and Owner

You first need to give your A/B test a name, select an assignment table, choose an entity, and assign an owner. Use a descriptive name that others understand. The assignment table determines which entity was exposed to which treatment. The A/B test uses the entity to randomly assign treatment and to aggregate metrics. For this exercise, use:
  • Name: aa-ab-test-<your-name-and-date>
  • Assignment table: Select an appropriate assignment table for your entity. This option only appears if your account has more than one assignment table.
  • Entity: Visitor
  • Owner: Select yourself
Click Create. You’re now on the A/B test configuration page.

​
Choose the Treatments

The treatments you choose for your A/B test decide what experiences you want to compare. The treatments you want to test must exist as variants on a feature flag. Feature flags define a configuration for an aspect of your app, website, or backend service. This step is where you select precisely what you want to vary in your A/B test.
The first treatment group you add is the control group. The experiment compares all other treatments to the first treatment. You can change which group is the control by dragging a group to the left-most position. Use the current default experience as the control variant.
In this guide, you don’t want to test a real feature or change for your users. Instead, you want to use the tutorial-feature flag. This flag changes nothing in your code, but only serves the purpose of testing the A/B test functionality. Click + Add control to add your first treatment group, which becomes the control. Then click + Add treatment to add a second treatment group. Select:
  • Flag: tutorial-feature
  • Control: control
  • Treatment: treatment

​
Define the Audience

The Audience section is where you define what your A/B test should target. The Randomization field determines what field in the evaluation context of the flag that your rollout randomizes treatment assignment on. Confidence pre-populates this field with the entity you selected in the creation step. The context schema maps the entity Visitor to the field ‘visitor_id’ (shown in parentheses). This means that the rollout randomizes on the value passed in the feature flag’s evaluation context field ‘visitor_id’. You decide the target audience for your A/B test. For example, add country is Sweden as an inclusion criterion if you want to target users in Sweden. The information available in the evaluation context of the flag determines what you can target on. Read more on the Audiences page. In this case, target everyone by leaving the inclusion criteria empty.
The evaluation context is information that you pass in when making the resolve call to Confidence to ask what variant to serve. This means what’s available in the evaluation context depends on what you pass in. Confidence lists recently used attributes available in the evaluation context when you enter the name of the attribute.
This video gives a quick overview of how to increase the flexibility of who you can include and exclude in your experiments by providing more information when you resolve feature flags in 2 minutes and 10 seconds.

​
Set the Allocation

The allocation sets what proportion of your target audience is eligible for your A/B test. Adjust the slider to 5% to allocate 5% of the traffic to the A/B test. You can also set the allocation by entering 5% in the input field.

​
Add Metrics

In this step you select the metrics you want to track for your A/B test. Two types of metrics are available for A/B tests. Success metrics are metrics you intend to improve with your treatment. Guardrail metrics are metrics you don’t expect to improve, but you want to make sure they don’t deteriorate. Use both types to decide if the treatment variant is better than control for your product.
Confidence uses the entity you selected in the creation step to decide which metrics you can select. Any metric based on the Visitor entity is available. You need to use a fact table that includes the Visitor entity if you want to create a new metric for it.

​
Add Success Metric

In the success metric section, click Add metric and select:
  • Metric: Page views per visitor metric.
  • Preferred direction: Increase. If you use a metric other than Page views per visitor, set this to the direction you want your metric to move in.
  • Minimum detectable effect (MDE): 5%.

​
Add Guardrail Metric

In the guardrail metric section, click Add metric and select:
  • Metric: Page views per visitor metric.
  • Preferred direction: Increase. If you use a metric other than Page views per visitor, set this to the direction you want your metric to move in.
  • Non-inferiority margin (NIM): 3%.

​
Write a Hypothesis

In this section, you should clearly describe your hypothesis so that everyone understands the purpose of the A/B test. See ideas for good hypotheses and learn more on the hypothesis page. In the Hypothesis section at the top of the configuration page, click the text area and enter your hypothesis. Hypothesis: Changing the tutorial-feature from the variant control to treatment for everyone should result in a change in their behavior, as measured by the success metric Page views per visitor. The data supports the hypothesis if the success metric improves by 5% and the guardrail metric Page views per visitor doesn’t deteriorate by more than 3%.

​
Calculate the Sample Size

In this step you run the sample size calculator to find out how many users your A/B test needs. This number represents how many users you need to have a reasonable chance of finding significant results if the treatment truly improves the user experience as much as you hope for. Read more about power analysis and the required sample size. Click Calculate in the Required sample size section on the right sidebar.
The sample size calculation queries historical data in the data warehouse to estimate the required sample size. This might take a few minutes.
If the required sample size is smaller than the number of users you can receive with the current allocation, your A/B test can reach power. To learn more about how to work with design and the statistical settings of an A/B test, see the Alpha and Power page in the documentation. You have successfully configured your A/B test, great job!

​
Launch the A/B Test

This step launches your A/B test. Make sure that you have selected the tutorial-feature flag to avoid changing a real experience.
MCP can’t launch A/B tests. Ask the Confidence Agent in the app or in Slack:
The agent asks you to confirm, then launches the A/B test.
If there are other live rollouts or A/B tests that use the tutorial-feature flag, you may not receive 5% of the traffic. Click Flags on the left sidebar and go to the tutorial-feature flag. In the Rules section, you see what rules exist on the flag. If other rules have higher priority than yours and use all traffic, you receive no traffic. If you want to receive traffic, you need to adjust the priority of your rule and move it up in the list. Read more about order of rules.
Congratulations, you have launched your first A/B test!

​
Monitor the A/B Test and Read Results

When you launch the A/B test, Confidence calculates exposure for the A/B test at repeated short intervals to make sure that the A/B test is working as expected, and that you are seeing some traffic. Keep the A/B test running and check back in tomorrow to see your first results.
The assistant uses the Experiments MCP tools list_experiments, get_experiment, and get_results. You can also ask the Confidence Agent in the app or in Slack to summarize the results.
Remember to end your A/B test within a couple of days to avoid wasting resources.

A/B Tests Reference

Deep dive into A/B test configuration and best practices

Statistical Settings

Configure significance levels and testing methodology

Hypothesis Design

Write effective hypotheses for your experiments