Skip to main content
On the surface page, you can configure settings that apply to all experiments on the surface.
Surface settings apply for experiments that run on the surface where the settings live. Only settings on the global surface apply to all experiments.

​
Reviews

Add suggested or required reviewers to all experiments on a surface. Required reviews block the launch of experiments until at least one of the required reviewers has approved the experiment setup. Suggested reviewers automatically show up as optional reviewers on the experiment design page.
Required reviews add friction to the experiment launch process. Use it only on surfaces where mistakes come with high costs.
Review requests appear as a todo-list item on the reviewer’s Confidence home page. To get Slack notifications, the reviewer needs to integrate their Slack account with Confidence.

​
Notifications

Send notifications for activities on a surface to Slack or email.

​
Exclusivity Groups

Use exclusivity groups in experiments to make them mutually exclusive to each other. You can configure exclusivity groups to be:
  • Optional: The exclusivity group is not automatically selected by experiments on the surface, but experimenters can choose to add it.
  • Suggested: The exclusivity group is pre-selected by experiments on the surface. Experimenters can choose to remove it.
To select an exclusivity group in an experiment, you need to select to run the experiment on the surface on which the exclusivity group lives. Read more about mutually exclusive experiments in the exclusivity documentation.
An exclusivity group is not allocating anything in itself. An exclusivity group is a resource that other experiments and rules can use to decide how to overlap or not overlap with each other.

​
Name Exclusivity Groups

Although exclusivity groups are naturally grouped by the surfaces they live on, you should give them names that are descriptive of the coordination you use them for. For example, if you create an exclusivity group to coordinate experiments for the search result ranker ML models, you can name the exclusivity group search-ranker.

​
Holdbacks

Holdbacks are a random set of users that you can reuse over time. Use holdbacks to:
  • Hold users back from a set of experiments for a certain period of time
  • Experiment on the same subset of users across several experiments over a period of time
You can configure holdbacks to be:
  • Optional: The holdback is not automatically selected by experiments on the surface, but experimenters can choose to add it.
  • Suggested: The holdback is pre-selected by experiments on the surface. Experimenters can choose to remove it.
  • Required: The holdback is pre-selected by experiments on the surface. Experimenters cannot choose to remove it.
If you create several holdbacks on the same surface, they are mutually exclusive to each other. Holdbacks on different surfaces are randomly overlapping. In one experiment, you can use one or several holdbacks from one or several surfaces.
Holdbacks don’t allocate any users themselves and Confidence collects no exposure or assignment data for units in the holdback. A holdback is a reference to a set of users that you can target or avoid in your experiments.

​
New Users in Holdbacks

A holdback is a random subset which makes up some proportion of all units. When new users come in, they spread proportionally into and outside of the holdback. For example, if you have a holdback with a 15% allocation, and 100 new users visit your app, around 15 of them belong to the holdback.

​
Metrics

Add metrics to a surface to make them easier to find for experimenters experimenting on this surface, or enforce that all experiments on the surface check this metric for regressions by making it required. Required metrics don’t have non-inferiority margins, but test for deterioration. These metrics don’t impact power analyses and sample size calculations. If an experiment is part of multiple surfaces, the required metrics are automatically de-duplicated. You can also explicitly add a required metric to an experiment. In this case, the configuration from the experimenter takes precedence, because Confidence checks all metrics that are part of an experiment for deterioration. Confidence alerts the user if any metric moves significantly in the non-desired direction. Read more about notifications for alerts in the notifications documentation.
Suggested metrics are shown at the top of the metrics list when experimenters select metrics for an experiment.

​
Actions

Trigger actions when experiments on a surface meet certain conditions. You can configure ‘if this then that’ rules. The following triggers are available:
  • Experiment ends
  • Metric moves significantly in the non-desired direction
With the following actions:
  • Create and run an exploratory analysis
  • End the experiment
For the actions based on metrics moving significantly in the non-desired direction, the metric must be a required metric on the surface. You can create multiple actions for the same trigger.
Example: If you want to end an experiment and create an exploratory analysis when a metric moves significantly in the non-desired direction, you create the following actions:
  • If metric <metric-name> moves significantly in the non-desired direction, then end the experiment
  • If metric <metric-name> moves significantly in the non-desired direction, then create and run an exploratory analysis with dimensions <dimension-name> and <dimension-name>

Surfaces Introduction

Learn about surfaces in Confidence

Organize Experiments

Coordinate experiments with surfaces

Exclusive Experiments

Configure mutually exclusive experiments

Reviews

Manage experiment reviews