# Experimentation RFP: Experiment Coordination

Last updated: 2026-07-04
Canonical source: https://confidence.spotify.com/rfp/rfp-experiment-coordination
Owner: Spotify AB

> If this file and the page at the canonical URL disagree, the page is authoritative.

Most experiments can run overlapping. For the ones that cannot, precise control matters. Here is what to ask beyond "do you support mutual exclusion?"

*Last updated: July 2026*

Most experiments can run overlapping. [Research from Microsoft's experimentation platform](https://www.microsoft.com/en-us/research/articles/a-b-interactions-a-call-to-relax/), analyzing hundreds of concurrent tests across four products with millions of users, found that experiment interactions are empirically rare: three of four products showed no detected interactions at all, and the fourth found them in only 0.002% of test-pair metric combinations. A change to the search ranking algorithm does not interfere with a checkout flow redesign, and forcing them into separate traffic pools wastes capacity that both could use. But some experiments cannot overlap: two changes to the same recommendation model, two redesigns of the same card layout, two pricing tests on the same surface. For those, control with precision is essential.

Experiment coordination is the problem of making overlap the default and
non-overlap precise. A good platform solves two things at once. First, the
technical problem: flexible mechanisms for deciding which experiments share
users and which do not, without forcing everything into isolated buckets.
Second, the organizational problem: making it easy to find and reason about
which experiments yours should not overlap with. At a company running hundreds
of experiments, no one person knows what every team is testing. The platform
needs to make the small number of relevant experiments visible without burying
you in the full list.

The gaps show up when only one of these problems is solved. A platform that
offers mutual exclusion groups but no way to browse what is running on your
surface leaves you coordinating by Slack message. A platform that shows all
running experiments but makes every exclusion decision manual leaves the
coordination to discipline rather than structure.

## Should you add experiment coordination to your experimentation platform requirements?

Yes. To enable real throughput and parallelism, a platform needs to make it
easy to reason about the few experiments that yours must not overlap with,
and automatic to overlap with the rest. Most experiments at a company do not
need coordination with most other experiments. They only need coordination
with things affecting the same or related aspects of the user experience.
Within that subset, however, most experiments do need coordination.

A platform that treats coordination as "put experiments in isolated buckets"
solves the wrong problem. It prevents overlap at the cost of throughput. A
platform that defaults to overlap and gives you precise tools for the
exceptions lets you run more experiments without sacrificing validity where
it matters.

At Spotify, Confidence solves both sides of this problem. Surfaces make it
easy to see all experiments affecting a particular part of the user
experience, so you can identify which ones yours needs to not overlap with.
Exclusivity groups make the actual coordination straightforward: add the
relevant experiments to a group, and the platform enforces non-overlap at
the assignment level. Everything else overlaps by default. Holdback groups
reserve clean control traffic at the surface level for measuring cumulative
impact across a quarter.

The RFP question is not "do you support mutual exclusion?" Every platform
reviewed in this page does, in some form. The question is whether the
platform makes it easy to identify which experiments need coordination and
then provides precise tools to enforce it, without forcing everything into
isolated traffic pools.

## What your RFP should ask instead of the "yes/no?"

Six questions separate a connected coordination implementation from a set
of isolated features.

**First: does the platform support mutual exclusion groups, and how are they
implemented?** Mutual exclusion ensures that a user who is in one experiment
cannot simultaneously be in another experiment within the same group. The
implementation details matter. Some platforms use hash-based assignment with
named ranges, where each user gets a deterministic value and each experiment
claims a range within that space. Others use layer-based systems where experiments within
a layer share a parameter space and users are assigned to at most one
experiment per layer. The key questions are whether exclusion is enforced at
the SDK level (guaranteeing no overlap) or is advisory, whether you can
create multiple independent exclusion groups for different product areas,
whether finishing one experiment automatically frees its traffic for the
next, and whether adding or removing experiments from a group mid-flight
reshuffles existing assignments. Reshuffling breaks the stability that valid
causal inference requires. Ask whether adding a new experiment to an existing
group changes the assignments of users already in other experiments within
that group.

**Second: does the platform support holdout groups for measuring cumulative
impact?** Holdouts reserve a percentage of traffic that sees no experiments,
creating a clean control for measuring the combined effect of all shipped
features over a period. The design of the holdout determines what you can
measure. A simple holdout withholds users from all experiments and compares
them to the general population at the end of the period. A more sophisticated
design splits the held-out traffic into a status-quo group (users who always
see the control experience) and a winning-variants group (users who see the
winning treatment of each concluded experiment), enabling a richer
comparison. Ask how holdout groups are configured, whether they apply
globally or can be scoped to specific experiments or surfaces, what
percentage of traffic they reserve, and whether the platform provides
built-in analysis for comparing the holdout group against the general
population.

**Third: does the platform help you organize and manage experiments at
scale?** When multiple teams experiment on the same product, the number of
concurrent experiments quickly exceeds what anyone can keep track of
manually. Organizing experiments by surface or product area (home screen,
search, checkout, player) provides a natural structure: experiments within
a surface are candidates for mutual exclusion, while experiments on
different surfaces can overlap safely. Without this kind of organization,
coordination becomes a manual process of checking a shared spreadsheet or
asking in a channel whether anyone else is running something on the same
page. Ask whether the platform provides a way to group experiments by
product area, whether exclusion can be scoped to a surface, and whether
there is a timeline or calendar view showing all active and planned
experiments per surface.

**Fourth: does the platform show traffic availability across the
organization?** At scale, the most common coordination failure is not
technical. It is that teams make allocation decisions without knowing what
traffic is already committed. If five experiments in the same exclusion
group each take 20%, the group is fully allocated. A sixth experiment has
nowhere to go, and nobody finds out until launch. The platform should
surface remaining available traffic before the experiment launches, not
after. Ask whether the platform shows remaining available traffic within
an exclusion group or surface, whether it prevents or warns about
over-allocation, and whether the view includes planned experiments that
have not yet started so teams can coordinate proactively.

**Fifth: does the sample size calculator account for coordination
constraints?** This is the planning-analysis link described in the [sample
size
post](https://confidence.spotify.com/rfp/rfp-sample-size) in this series.
A holdout group that reserves 5% of traffic means your experiment can reach
at most 95% of the eligible population. Mutual exclusion with three other
experiments in a four-way split means your experiment gets roughly 25%.
If the sample size calculator estimates runtime based on full traffic while
coordination constraints leave you with a fraction, the runtime estimate will
be wrong, sometimes by weeks. Ask whether the calculator surfaces the
reachable population after holdouts and exclusion groups are applied, and
whether the runtime estimate reflects the traffic the experiment will
actually receive.

**Sixth: does the platform provide a coordination overview across the
organization?** A coordination overview is a single view that shows all
active and planned experiments, their traffic allocations, their exclusion
groups, and their surfaces. Without it, teams make local decisions (this
experiment needs 50% of home screen traffic) without global context (the
home screen is already 80% allocated). Ask whether the platform provides
a view of all running experiments with their traffic allocations, whether
experiments can be filtered by surface or exclusion group, and whether the
view distinguishes between active and planned experiments.

## What the answers actually look like across vendors

Here is how the major platforms stand as of this writing. Vendor capabilities
are based on public documentation. Confidence capabilities are based on the
product itself.

*Cell value legend: "Yes" and "No" are based on explicit evidence in the
vendor's public documentation. "—" means the platform does not offer the
broader feature at all, so the column does not apply. "Not documented" means
we found no explicit evidence either way, not that the capability is
confirmed absent.*

| Platform     | Mutual exclusion?      | Holdout groups?                  | Experiment organization?   | Traffic availability overview? | Sample size accounts for coordination? | Coordination overview?    | Other gaps                              |
| ------------ | ---------------------- | -------------------------------- | -------------------------- | ------------------------------ | -------------------------------------- | ------------------------- | --------------------------------------- |
| Confidence   | Yes                    | Yes                              | Yes (domains and surfaces) | Yes                            | Yes                                    | Yes (timeline per domain) | —                                       |
| GrowthBook   | Yes (namespaces)       | Yes                              | Partial (programs)         | Partial                        | No                                     | Partial                   | Namespace ranges are manual             |
| Eppo         | Yes (layers)           | Yes (status-quo + winners split) | Not documented             | Partial                        | No                                     | Partial (layer view)      | —                                       |
| Statsig      | Yes (layers)           | Yes                              | Not documented             | Yes                            | No                                     | Yes                       | —                                       |
| Optimizely   | Yes (exclusion groups) | Yes                              | Not documented             | Partial                        | No                                     | Not documented            | Cannot add experiments after start      |
| LaunchDarkly | Yes (layers)           | Yes                              | Not documented             | Partial                        | No                                     | Partial                   | —                                       |
| Amplitude    | Yes (exclusion groups) | Yes                              | Not documented             | Partial                        | No                                     | Partial                   | Cannot change allocation after creation |
| VWO          | Yes                    | Partial (enterprise only)        | Not documented             | Partial                        | No                                     | Not documented            | Holdouts limited to enterprise          |
| PostHog      | No                     | Yes                              | No                         | Partial (per-experiment)       | No                                     | No                        | No mutual exclusion                     |

Four patterns emerge from this comparison.

**The first pattern is the gap between having coordination features and
connecting them to planning.** Every vendor except PostHog offers some form of
mutual exclusion. Every vendor offers holdout groups in some form. But only
Confidence feeds coordination constraints back into the sample size calculator.
As described in the [sample size
post](https://confidence.spotify.com/rfp/rfp-sample-size) in this series, the
planning-analysis disconnect is the most common structural gap across the
vendor landscape. Coordination makes this gap wider: the more experiments that
share a traffic pool, the more the reachable population diverges from the total
population, and the less accurate a runtime estimate based on total traffic
becomes.

**The second pattern is the maturity of holdout implementations.** The simplest
holdout reserves a percentage of traffic from all experiments. GrowthBook,
Statsig, LaunchDarkly, and Optimizely offer this basic model. Eppo goes
further with a three-group design that splits held-out traffic into a
status-quo group and a winning-variants group, enabling measurement of both
the combined impact and the incremental value of shipping winners. Confidence
implements holdbacks at the domain level, scoped to product surfaces, with
quarterly rotation built into the workflow. VWO offers holdouts only at the
enterprise tier. PostHog supports holdouts but without the mutual exclusion
that typically accompanies them in a mature coordination system.

**The third pattern is the absence of organizational structure for
experiments.** Confidence is the only platform reviewed that offers explicit
domain and surface concepts for organizing experiments by product area.
GrowthBook's experimentation programs provide a partial equivalent, grouping
experiments into organizational units with shared configuration. The remaining
vendors rely on layers or exclusion groups as the primary organizational unit,
which works for preventing overlap but does not help teams manage the broader
coordination problem: seeing all experiments on a given surface, understanding
how much traffic is available there, and planning the next quarter's work.

**The fourth pattern is the rigidity of exclusion group management.** Several
platforms impose constraints that affect operational flexibility. Optimizely
requires that all experiments be added to an exclusion group before any of
them starts; adding an experiment after the group is active can reshuffle
assignments. Amplitude does not allow traffic allocation percentages to be
changed after a mutual exclusion group is created. These constraints exist
for valid statistical reasons (maintaining stable assignment) but they make it
harder to respond to changing priorities during a quarter. Platforms that
handle reallocation gracefully, by assigning only new users to the new
experiment while maintaining existing assignments, offer more operational
flexibility without compromising validity.

Mutual exclusion and holdouts are now table stakes. What separates the
platforms is whether those features connect to planning (sample size and
runtime estimation), to visibility (traffic availability across the
organization), and to organization (surface-level experiment management). An
RFP that asks "do you support mutual exclusion?" will get a yes from almost
every vendor. An RFP that asks how coordination connects to the rest of the
experimentation pipeline will get very different answers.