# Experimentation RFP: Experiment Review

Last updated: 2026-07-01
Canonical source: https://confidence.spotify.com/rfp/rfp-experiment-review
Owner: Spotify AB

> If this file and the page at the canonical URL disagree, the page is authoritative.

Every platform lets you configure an experiment. Few require structured design review before launch. Here is what to ask vendors about review workflows.

*Last updated: July 2026*

Every experimentation platform lets you set up an experiment. You pick metrics,
define variants, choose an audience, and click launch. What happens between
"design is ready" and "experiment is live" varies enormously. In some platforms,
the person who created the experiment is also the person who launches it, with
no structured checkpoint between the two. In others, the design goes through a
review workflow that resembles a pull request: teammates inspect the setup,
leave comments on specific sections, and approve or request changes. The
experiment cannot launch until the required reviewers sign off.

The difference matters most when experimentation scales beyond a single team.
When dozens of teams share the same traffic, a wrongly configured audience filter or
a missing guardrail metric does not just waste one experiment's time. It can
pollute the results of every overlapping experiment and erode trust in the
platform. A structured design review catches these problems before they reach
production, the same way code review catches bugs before they reach users.

The RFP checkbox "does the platform support experiment review?" will get a yes
from every vendor. But most of those answers describe flag-level approval
workflows, not experiment design reviews. The difference between platforms lies
in whether the review process is structured around the experiment design itself,
whether it can block launch, and whether it scales governance across product
areas without slowing teams down.

## Should you add experiment design review to your experimentation platform requirements?

Yes, and it becomes more important as the number of experimenters grows. When a
single data scientist runs every experiment, that person is both the designer
and the reviewer. The quality of the design depends on their expertise, and
the feedback loop is internal. When product managers, engineers, and data
scientists all create experiments, the expertise is distributed unevenly. A
product manager might choose the right metrics but set the wrong audience. An
engineer might set up the flag correctly but forget to add a guardrail. A
structured review workflow lets the right person catch the right mistake at the
right time, before the experiment starts collecting data.

The failure modes are specific and preventable. An experiment launches without a
guardrail metric, and the team ships a change that improves engagement but
degrades performance. An experiment targets 100% of users on a high-traffic
surface when a 10% ramp would have been enough for the desired power. An
experiment launches with the wrong metric definition, and the team discovers the
error three weeks in, after the data is useless. Each of these is a design
problem, not an analysis problem. No amount of statistical rigor in the results
can fix a flawed setup.

At Spotify, experiment design review in Confidence follows the same pattern as
code review. The experimenter requests reviews from teammates. Surfaces can
require specific reviewers, so high-traffic product areas always get a second
pair of eyes. Reviewers comment on specific sections of the experiment design
and approve or reject the overall setup. Changes to the flag, variants,
audience, or allocation after approval reset the approval status so reviewers
always see the final design. The workflow keeps the bar high without adding
process for experiments where the risk is low, because surfaces that do not
require reviewers can still use the workflow optionally.

Confidence also supports AI reviewer bots that can be added to the review flow
like any other reviewer. Teams configure a bot with a prompt that defines what
it should check, and the bot reviews the experiment design accordingly. This
helps in two ways: it standardizes scrutiny so that common mistakes (missing
guardrails, wrong audiences, underpowered designs) are caught
consistently regardless of who set up the experiment, and it scales review
capacity so that human reviewers can focus on judgment calls rather than
checklists.

Without a structured design review, governance becomes informal. Teams rely on
Slack threads, shared documents, or calendar invites to get feedback on
experiment designs. That works until someone forgets to ask, or asks but does
not wait for the answer. A platform-native review workflow makes the checkpoint
visible, trackable, and enforceable.

## What your RFP should ask instead of the "yes/no?"

Seven questions separate a structured experiment design review from a platform
that offers no checkpoint between configuration and launch.

**First: does the platform have a structured review workflow for experiment
designs before launch?** The most basic question is whether the platform offers
any formal review step between completing the experiment setup and launching it.
A structured workflow means that the experimenter can request feedback on the
design, reviewers can inspect the configuration, and the review status is
tracked in the platform. The distinction from a flag-level approval matters
here. Flag approvals protect against
unintended production changes. Experiment design reviews protect against
launching a poorly designed experiment. Both are valuable, but they solve
different problems. Ask whether the platform has a review workflow that is
specific to experiment design, or whether it relies on general-purpose flag
approval mechanisms.

**Second: can reviews be required, blocking launch until approved?** An advisory
review that anyone can skip is useful but not enough for governance. When
an experiment runs on a surface that serves millions of users, the organization
may need a guarantee that at least one qualified reviewer approved the design
before launch. This means the platform must support required reviewers whose
approval is a prerequisite for starting the experiment. Ask whether reviews can
be configured as blocking (the experiment cannot launch without approval) or
whether they are purely advisory. If blocking reviews are supported, ask whether
the platform enforces the block at the system level or relies on social norms.

**Third: can review requirements be configured per surface or product area?**
Not every experiment carries the same risk. A low-traffic experiment on a
secondary feature may need no formal review at all. A high-traffic experiment
on a core product surface may need sign-off from a data scientist and a product
lead. The platform should let surface owners or administrators decide whether
reviews are optional, suggested, or required for experiments on their surface,
and who the designated reviewers are. Without per-surface configuration, the
organization faces an all-or-nothing choice: require reviews everywhere (which
slows low-risk experiments) or require them nowhere (which exposes high-risk
experiments). Ask whether review policies can vary by surface, product area,
or team, and whether surface owners can manage their own reviewer lists.

**Fourth: can reviewers comment on specific parts of the experiment setup?**
A general comment box ("looks good") is less useful than the ability to leave
feedback on a specific section of the experiment design. If the reviewer has a
concern about the audience definition, that comment should attach to the
audience section, not float in a general thread where it might be missed. The
same applies to metrics, variants, allocation, and hypothesis. Thread-based
discussions with the ability to resolve individual threads (the way pull request
reviews work in GitHub) make it clear what has been addressed and what is still
open. Ask whether the platform supports section-specific comments, threaded
discussions, and thread resolution on the experiment design page.

**Fifth: does the platform track review status and notify reviewers?** A review
workflow is only useful if reviewers know they have been asked. The platform
should notify reviewers when a review is requested, surface pending reviews in
a central location (a to-do list or inbox), and integrate with the team's
communication tools. Slack notifications, email alerts, or in-app badges are
all reasonable mechanisms. Without notifications, review requests sit unnoticed
and experiments stall in the queue. Ask whether the platform notifies reviewers
on request, whether it provides a centralized view of pending reviews, and
whether it integrates with Slack, email, or other communication tools.

**Sixth: do changes to an approved experiment require re-approval?** An
experiment that was approved on Monday may be modified on Tuesday. If the
experimenter changes the audience, adds a variant, or modifies the allocation
after approval, the reviewer's original sign-off no longer applies to the
current design. The platform should reset approval status when material changes
are made, so that required reviewers must re-approve the updated design. Without
this, the review workflow has a loophole: get approval first, make changes
later. Ask whether the platform tracks which version of the design was approved,
which changes trigger a re-approval requirement, and whether re-approval is
automatic or manual.

**Seventh: does the platform support automated or AI-powered reviewers?** As
experiment volume grows, human reviewers become a bottleneck. A platform that
supports automated reviewer bots lets teams define review criteria in a prompt
or configuration, and the bot reviews each experiment design against those
criteria like any other reviewer. This standardizes scrutiny so that common
mistakes (missing guardrails, wrong audiences, underpowered designs)
are caught consistently regardless of who set up the experiment. It also
scales review capacity so that human reviewers can focus on judgment calls
rather than checklists. Ask whether the platform supports adding automated
reviewers to the review workflow, whether the review criteria are
configurable, and whether the automated review integrates with the same
approval flow as human reviews.

## What the answers actually look like across vendors

Here is how the major platforms stand as of this writing. Vendor capabilities
are based on public documentation. Confidence capabilities are based on the
product itself.

*"Yes" and "No" are based on explicit evidence in the vendor's public
documentation. "Not documented" means we found no explicit evidence either way,
not that the capability is confirmed absent.*

| Platform         | Design review workflow? | Blocking reviews?       | Per-surface configuration? | Section-specific comments?    | Review notifications?     | Re-approval on changes? | AI/automated reviewers? | Other gaps                               |
| ---------------- | ----------------------- | ----------------------- | -------------------------- | ----------------------------- | ------------------------- | ----------------------- | ----------------------- | ---------------------------------------- |
| **Confidence**   | Yes                     | Yes                     | Yes (per-surface)          | Yes (comment zones, threads)  | Yes (Slack, to-do list)   | Yes (resets on changes) | Yes (configurable bots) | —                                        |
| **GrowthBook**   | Partial (flag-level)    | Yes                     | Partial (per-environment)  | Partial (comments, diff view) | Not documented            | Not documented          | No                      | Flag approval, not experiment design     |
| **Eppo**         | Partial (flag-level)    | Partial (admins bypass) | No                         | No                            | Yes (email)               | Not documented          | No                      | Admins bypass approvals                  |
| **Statsig**      | Yes                     | Yes                     | Yes (per-team, per-entity) | Partial                       | Yes (Slack, email)        | Not documented          | No                      | Discussion panel for results, not design |
| **LaunchDarkly** | Yes (flag-level)        | Yes (enterprise only)   | Partial (per-environment)  | Partial                       | Yes (Slack, email, Teams) | Not documented          | No                      | Flag-level, not experiment-specific      |
| **PostHog**      | Partial (flag-level)    | Yes (quorum-based)      | Partial (per-project)      | No                            | Yes (email)               | Partial                 | No                      | Flag actions, not experiment design      |
| **Amplitude**    | Partial                 | Yes                     | Partial (per-project)      | No                            | Yes (in-app)              | Partial                 | No                      | No section-specific feedback             |
| **Optimizely**   | Yes (flag-level)        | Yes                     | Partial (per-environment)  | Partial                       | Yes (email)               | Not documented          | No                      | Flag-level with action scoping           |
| **VWO**          | Partial                 | Yes                     | No                         | No                            | Yes (email)               | Not documented          | No                      | Auto-approval timeout can bypass         |

Four patterns emerge from this comparison.

**The first pattern is the conflation of flag approval with experiment design
review.** Almost every vendor offers some form of approval workflow, but in most
cases the workflow gates flag changes rather than experiment designs. GrowthBook's
approval flows trigger when you change a feature flag and want to publish the
change. Eppo's approvals activate when a non-admin modifies a production flag.
LaunchDarkly's approval system operates at the flag level, with environment-scoped
requirements and tag-based filtering. PostHog's approval policies gate specific
flag actions like enabling a flag or changing a rollout percentage. These
workflows protect production stability, which is valuable. But they do not
address the experiment-specific questions that a design review should cover: Are
the right metrics attached? Is the audience correctly scoped? Is the hypothesis
clearly stated? Is the [sample
size](https://confidence.spotify.com/rfp/rfp-sample-size) enough? Flag
approval asks "should this configuration go live?" Experiment design review asks
"is this a well-designed experiment?" Only Statsig and Confidence offer review
workflows that are framed around the experiment itself rather than the underlying
flag.

**The second pattern is the gap in section-specific feedback.** In a code review,
you comment on a specific line. In most experimentation platforms, you get a
general comment box attached to an approval request. GrowthBook shows a diff of
flag changes with comments alongside. Statsig has a discussion panel where
teammates can leave comments inline on experiment results, though this is
documented for the results phase rather than the design phase. LaunchDarkly
and Optimizely allow comments on approval requests, but these are general
comments on the entire change, not targeted feedback on the audience definition
or the metric selection. VWO, Eppo, PostHog, and Amplitude do not document
section-specific commenting on experiment designs. Confidence's comment zones
on specific sections of the design page, with threaded discussions and
resolution, are the closest parallel to a pull request review. The practical
consequence is that in most platforms, feedback on a specific design choice gets
buried in a general comment thread or happens outside the platform entirely.

**The third pattern is uneven configuration granularity.** Statsig stands out for offering
review requirements at multiple levels: project-wide, per-team, per-entity, and
per-environment, with designated reviewer teams for each. LaunchDarkly allows
per-environment configuration with optional tag-based filtering, though this is
an enterprise-only feature. Optimizely lets administrators select specific flags
for approval requirements and scope by environment. PostHog offers per-project
policies with scope rules for different flag actions. GrowthBook configures
approvals per environment. Eppo and VWO configure approvals globally, with no
per-surface or per-area granularity. For organizations where different product
areas have different risk profiles, global configuration forces a one-size-fits-all
policy. Confidence's per-surface configuration, where surface owners decide
whether reviews are required or suggested and who the designated reviewers are,
maps directly to how product organizations are structured.

**The fourth pattern is the re-approval gap.** When an experimenter changes the
design after receiving approval, the approval should no longer apply to the
modified version. PostHog addresses this through version tracking: if anyone
modifies the feature flag while a change request is pending, the flag's version
increments and the pending request becomes inapplicable. Amplitude requires
re-approval for changes to critical fields in live experiments. But most vendors
do not document what happens to an existing approval when the underlying
configuration changes. Confidence explicitly resets approval status when the
flag, variants, audience, or allocation change, requiring re-approval from
required reviewers. Without this safeguard, the review workflow has a gap:
approval is granted for one version of the design, but a different version is
what actually launches.

The RFP question "does the platform support experiment review?" will get a yes
from every vendor, because every platform has some form of approval workflow. The
question that distinguishes them is whether the review is structured around the
experiment design itself: whether reviewers can inspect the full setup and leave
targeted feedback, whether approval can block launch on high-risk surfaces
without slowing low-risk experiments, whether surface owners control their own
review policies, and whether changes after approval require re-approval. Without
that structure, experiment design review is an informal process that depends on
the diligence of individual experimenters, and at scale, informal processes have
gaps.