Gradually Release Your Changes
A rollout is a type of experiment that lets you gradually introduce a change to users instead of releasing it immediately to everyone. A rollout enables monitoring of the effect of the change on important metrics to make sure that the change is working correctly when releasing it. If needed, you can roll back the change instantly without having to do a new release. In summary, a rollout gives you:- Monitoring of a set of metrics that you choose to track, including business metrics, performance metrics, and user-behavior metrics
- Alerts if any of your metrics move in the wrong direction or if there’s a sign that there’s a problem with your setup
- Recommendations for how to proceed with the rollout
This guide requires a feature flag and at least one metric. In what follows, the guide
uses the flag
tutorial-feature with a treatment variant, and the metric Page views per visitor for the
Visitor entity. Use other flags, variants, metrics and entities whenever the guide refers to
these if yours have different names. If necessary, follow the
flags quickstart to create the tutorial-feature flag, and
the metrics quickstart to create the Page views per visitor metric.Step 1: Create a Rollout
Time to get started and create your rollout. Open Confidence and select Rollout on the left sidebar. The overview page shows all draft, live, and ended rollouts that you have permission to view. Click + Create in the upper right corner to create a new rollout.Step 2: Name, Entity, and Owner
You first need to give your rollout a name and assign an owner. Use a descriptive name that others understand. You need to decide the entity that you want to rollout to. The rollout uses the entity to randomly assign treatment and to aggregate metrics. For this exercise, use:- Name:
aa-rollout-<your-name-and-date> - Entity: Visitor
- Owner: Select yourself
Step 3: Feature
The feature you choose for your rollout determines what you want to roll out. The feature you want to release must exist as a variant on a feature flag. Feature flags define a configuration for an aspect of your app, website, or backend service. This step is where you select precisely what you want to control in your rollout. In this guide, you don’t want to roll out a real feature or change for your users. Instead, you want to use thetutorial-feature flag. This flag changes nothing in your code, but only
serves the purpose of testing the rollout functionality.
Click add variant and select:
- Feature flag:
tutorial-feature - Variant:
treatment
The variant selected here is the variant you want to roll out. Before you reach 100% with your
rollout, not everyone receives the new variant. The users that don’t receive the new feature instead
get their variant set from other experiments, flag rules or the defaults defined by the clients
that use the flag.
Step 4: Audience
The Audience section is where you define what your rollout should target. Thetreatment randomization unit field determines what field in the evaluation
context of the flag that your rollout randomizes treatment assignment on.
Confidence pre-populates this field with the entity you selected in the creation
step. The context schema maps the entity ´Visitor´ to
the field visitor_id (shown in parenthesis). This means that the rollout
randomizes on the value passed in the feature flag’s evaluation context field visitor_id.
You decide the target audience for your rollout. For example, add
country is Sweden as an inclusion criterion if you want to target users in Sweden. The information
available in the evaluation context of the flag determines what you can target on. Read more on the
Audiences page.
In this case, target everyone and leave the inclusion criteria empty.
The evaluation context is information that you pass in when making the resolve call to Confidence to
ask what variant to serve. This means what’s available in the evaluation context depends on what you
pass in. Confidence lists recently used attributes available in the evaluation context when you enter the
name of the attribute.
Step 5: Metrics
In this step you select the metrics you want to track to make sure that your rollout isn’t negatively affecting the user experience or performance in any way. Confidence calls such metrics guardrail metrics.Confidence uses the entity you selected in the creation step to decide which metrics you can select. Any metric based on the Visitor entity is available.
You need to use a fact table that includes the Visitor entity if you want to create a new metric for it.
- Metric:
Page views per visitormetric. - Non-desired direction:
Decrease. If you use a metric other thanPage views per visitor, set this to the direction you don’t want your metric to move in. - Non-inferiority margin: 0.
Step 6: Launch
Now it’s time to launch your rollout! Click Launch in the top right corner.Step 7: Scale Up
When you’ve launched your rollout, you can scale up the reach to what you want. Use the slider to scale it up to 5%. Click Save for the new reach to apply. You’re now serving thetreatment variant of the tutorial-feature flag to 5% of your users.
If there are other live rollouts or A/B tests that use the
tutorial-feature flag, you may not receive
5% of the traffic. Click Flags on the left sidebar and go to the tutorial-feature flag. In the
Rules section, you see what rules exist on the flag. If other rules have higher priority than
yours and use all traffic, you receive no traffic. If you want to receive traffic, you need
to adjust the priority of your rule and move it up in the list. Read more about
order of rules.Step 8: Monitoring and Results
When you launch the rollout, Confidence calculates exposure for the rollout at repeated short intervals to make sure that the rollout is working as expected, and that you are seeing some traffic. Hover theLive status on the
right sidebar to see the current status of the checks run for the rollout.
You can end the rollout by clicking End in the upper right corner. Keep it running for longer
and check back in tomorrow to see your first results.
Remember to end your rollout within a couple of days to waste resources.