Skip to main content
By the end of this guide, you have a live rollout that serves a change to 5% of your users while Confidence monitors a guardrail metric.
Do it with AI. Ask the Confidence Agent in the app or in Slack:
The agent creates the rollout. It launches it only after you confirm. Review the configuration in the app before you launch.

​
Gradually Release Your Changes

A rollout is a type of experiment that lets you gradually introduce a change to users instead of releasing it immediately to everyone. A rollout enables monitoring of the effect of the change on important metrics to make sure that the change is working correctly when releasing it. If needed, you can roll back the change instantly without having to do a new release. In summary, a rollout gives you:
  • Monitoring of a set of metrics that you choose to track, including business metrics, performance metrics, and user-behavior metrics
  • Alerts if any of your metrics move in the wrong direction or if there’s a sign that there’s a problem with your setup
  • Recommendations for how to proceed with the rollout
Some sections link to associated videos and documentation. Make use of these resources as they contain important information that the guide doesn’t cover.

​
Before You Begin

This guide requires a feature flag and at least one metric. The guide uses the flag tutorial-feature with a treatment variant, and the metric Page views per visitor for the Visitor entity. If yours have different names, use your own flags, variants, metrics, and entities wherever the guide refers to these.
The assistant uses the Flags MCP tools createFlag, addFlagVariant, listEntities, and listMetrics. To create the metric with AI, see Create a metric.

​
Create a Rollout

MCP can’t create rollouts. Ask the Confidence Agent in the app or in Slack:
The agent can also set the feature, audience, and metrics from the next sections in the same request. See the prompt in the Do it with AI box at the top of this page.

​
Set the Name, Entity, and Owner

You first need to give your rollout a name and assign an owner. Use a descriptive name that others understand. You need to decide the entity that you want to roll out to. The rollout uses the entity to randomly assign treatment and to aggregate metrics. For this exercise, use:
  • Name: aa-rollout-<your-name-and-date>
  • Entity: Visitor
  • Owner: Select yourself
Click Create. You’re now on the rollout configuration page.

​
Choose the Feature

The feature you choose for your rollout determines what you want to roll out. The feature you want to release must exist as a variant on a feature flag. Feature flags define a configuration for an aspect of your app, website, or backend service. This step is where you select precisely what you want to control in your rollout. In this guide, you don’t want to roll out a real feature or change for your users. Instead, you want to use the tutorial-feature flag. This flag changes nothing in your code, but only serves the purpose of testing the rollout functionality. Click Add variant and select:
  • Feature flag: tutorial-feature
  • Variant: treatment
The variant selected here is the variant you want to roll out. Before you reach 100% with your rollout, not everyone receives the new variant. The users that don’t receive the new feature instead get their variant set from other experiments, flag rules or the defaults defined by the clients that use the flag.

​
Define the Audience

The Audience section is where you define what your rollout should target. The treatment randomization unit field determines what field in the evaluation context of the flag that your rollout randomizes treatment assignment on. Confidence pre-populates this field with the entity you selected in the creation step. The context schema maps the entity Visitor to the field visitor_id (shown in parenthesis). This means that the rollout randomizes on the value passed in the feature flag’s evaluation context field visitor_id. You decide the target audience for your rollout. For example, add country is Sweden as an inclusion criterion if you want to target users in Sweden. The information available in the evaluation context of the flag determines what you can target on. Read more on the Audiences page. In this case, target everyone and leave the inclusion criteria empty.
The evaluation context is information that you pass in when making the resolve call to Confidence to ask what variant to serve. This means what’s available in the evaluation context depends on what you pass in. Confidence lists recently used attributes available in the evaluation context when you enter the name of the attribute.
This video gives a quick overview of how targeting and evaluation contexts work in 2 minutes and 10 seconds.

​
Add Guardrail Metrics

In this step you select the metrics you want to track to make sure that your rollout isn’t negatively affecting the user experience or performance in any way. Confidence calls such metrics guardrail metrics.
Confidence uses the entity you selected in the creation step to decide which metrics you can select. Any metric based on the Visitor entity is available. You need to use a fact table that includes the Visitor entity if you want to create a new metric for it.
In the metric section, click Add metric and select:
  • Metric: Page views per visitor metric.
  • Non-desired direction: Decrease. If you use a metric other than Page views per visitor, set this to the direction you don’t want your metric to move in.
  • Non-inferiority margin: 0.
You have successfully configured your rollout.

​
Launch the Rollout

This step launches your rollout. Make sure that you have selected the tutorial-feature flag to avoid changing a real experience.
MCP can’t launch rollouts. Ask the Confidence Agent in the app or in Slack:
The agent asks you to confirm, then launches the rollout.

​
Scale Up the Reach

When you’ve launched your rollout, you can scale up the reach to what you want.
Ask the Confidence Agent in the app or in Slack:
The agent changes the reach of a live rollout when you ask. It can also set an automated ramp schedule, a series of reach steps that each last a set duration. Before the agent raises reach above the limit where monitoring stays on, it warns you that monitoring turns off permanently.
You’re now serving the treatment variant of the tutorial-feature flag to 5% of your users.
If there are other live rollouts or A/B tests that use the tutorial-feature flag, you may not receive 5% of the traffic. Click Flags on the left sidebar and go to the tutorial-feature flag. In the Rules section, you see what rules exist on the flag. If other rules have higher priority than yours and use all traffic, you receive no traffic. If you want to receive traffic, you need to adjust the priority of your rule and move it up in the list. Read more about order of rules.
Congratulations, you have launched your first rollout!

​
Monitor the Rollout and Read Results

When you launch the rollout, Confidence calculates exposure for the rollout at repeated short intervals to make sure that the rollout is working as expected, and that you are seeing some traffic. Keep the rollout running and check back in tomorrow to see your first results.
The assistant uses the Experiments MCP tools list_experiments, get_experiment, and get_results. You can also ask the Confidence Agent in the app or in Slack.
Remember to end your rollout within a couple of days to avoid wasting resources.

​
Bonus Step: Automate the Ramp-up of Your Rollout

Confidence lets you automate the ramp-up of your rollout. The rollout schema allows you to define the reach steps that the rollout should take and at what time points the increases in reach should happen. Read more about it in the rollout documentation.

Rollouts Reference

Deep dive into rollout configuration and automation

Launch an A/B Test

Compare variants with a controlled experiment

Configure Metrics

Set up metrics to monitor your rollout