Skip to main content
Beta
AI review agents are currently in beta. If you don’t see the Agents page in your organization, contact experimentation-cs@spotify.com to request access.
Create an AI review agent to provide automated feedback on A/B test and rollout designs. Agents analyze experiment configurations and post comments on individual sections based on your instructions.

​
Create the Agent

1

Go to Agents

Go to Confidence and select Admin on the left sidebar, then select Agents.
2

Create agent

Click Create and enter a name and description for your agent.
3

Enable Review skill

Toggle on the Review skill to allow the agent to review experiments.
4

Add instructions

Enter instructions that define how the agent should review experiments. See Write effective instructions for guidance.
5

Save

Click Save to create your agent.

​
Write Effective Instructions

Structure your instructions by section to help the agent provide targeted feedback. The agent reviews different sections depending on the experiment type. Map your instructions to the sections on the experiment Design page.

​
A/B Test Sections

SectionWhat to define
Display nameNaming conventions or required prefixes
HypothesisWhat makes a good hypothesis statement
FlagFlag selection and configuration expectations
TreatmentsRequirements for treatment descriptions, images, and allocation
AudienceTargeting criteria and allocation expectations
SurfacesSurface selection requirements
MetricsMetric selection guidelines, role assignments, MDE and NIM requirements, preferred directions
StatsAlpha, power, test horizon strategy, and exposure filter expectations
Sample sizeSample size calculation requirements
PlanningExperiment duration and scheduling expectations
LinksRequired documentation or resources

​
Rollout Sections

SectionWhat to define
Display nameNaming conventions or required prefixes
DescriptionWhat the rollout description should include
FeatureVariant selection and initial reach expectations
AudienceTargeting criteria and allocation expectations
SurfacesSurface selection requirements
MetricsMonitoring metric selection guidelines
StatsAlpha and power expectations
Sample sizeSample size calculation requirements
PlanningRollout duration and scheduling expectations
Automatic ramp-upRamp-up schedule, step count, and timing expectations
LinksRequired documentation or resources
Be specific about what the agent should check. For example, instead of “check the hypothesis,” write “verify the hypothesis states a clear expected outcome with a measurable effect on the primary metric.”

​
Preview Agent Reviews

Test how your agent reviews experiments before using it on real experiments.
1

Open agent

Go to Admin > Agents and select your agent.
2

Start preview

Click Preview.
3

Select experiment

Select a draft A/B test or rollout from the dropdown list.
4

Run review

Click Review to see how the agent would review the experiment.
The preview shows the agent’s overall judgment (approved or rejected) along with feedback for each section. Use this to refine your instructions.

​
Request an Agent Review

Request a review from an agent on the experiment Design page.
1

Open experiment

Go to an A/B test or rollout.
2

Add reviewer

In the Reviews section on the right sidebar, click the plus icon.
3

Select agent

Search for and select the review agent.
4

Request review

Click Request next to the agent name.
The agent analyzes the experiment and posts comments on individual sections where it has feedback.
AI agent reviews are informational and don’t block launching experiments. The agent provides guidance, but the final decision to launch rests with human reviewers.

Reviews

Learn about the review process

Configure Surface Reviews

Set up required reviewers for surfaces