Skip to main content
Beta
AI review agents are currently in beta. If you don’t see the Agents page in your organization, contact experimentation-cs@spotify.com to request access.
Create an AI review agent to provide automated feedback on A/B test and rollout designs. Agents analyze experiment configurations and post comments on individual sections based on your instructions.

​
Create the Agent

1

Go to Agents

Go to Confidence and select Admin on the left sidebar, then select Agents.
2

Create agent

Click Create and enter a name and description for your agent.
3

Enable Review skill

Toggle on the Review skill to allow the agent to review experiments.
4

Add instructions

Enter instructions that define how the agent should review experiments. See Write effective instructions for guidance.
5

Save

Click Save to create your agent.

​
Write Effective Instructions

Structure your instructions by section to help the agent provide targeted feedback. The agent reviews different sections depending on the experiment type. Map your instructions to the sections on the experiment Design page.

​
A/B Test Sections

​
Rollout Sections

Be specific about what the agent should check. For example, instead of “check the hypothesis,” write “verify the hypothesis states a clear expected outcome with a measurable effect on the primary metric.”

​
Preview Agent Reviews

Test how your agent reviews experiments before using it on real experiments.
1

Open agent

Go to Admin > Agents and select your agent.
2

Start preview

Click Preview.
3

Select experiment

Select a draft A/B test or rollout from the dropdown list.
4

Run review

Click Review to see how the agent would review the experiment.
The preview shows the agent’s overall judgment (approved or rejected) along with feedback for each section. Use this to refine your instructions.

​
Request an Agent Review

Request a review from an agent on the experiment Design page.
1

Open experiment

Go to an A/B test or rollout.
2

Add reviewer

In the Reviews section on the right sidebar, click the plus icon.
3

Select agent

Search for and select the review agent.
4

Request review

Click Request next to the agent name.
The agent analyzes the experiment and posts comments on individual sections where it has feedback.
AI agent reviews are informational and don’t count toward required reviews. An AI agent approval does not meet the required reviewer requirement for launching—at least one human reviewer must approve the experiment.

Reviews

Learn about the review process

Configure Surface Reviews

Set up required reviewers for surfaces