# Risk-Aware Product Decisions in A/B Tests with Multiple Metrics

Last updated: 2026-05-12
Canonical source: https://confidence.spotify.com/blog/risk-aware-decisions
Owner: Spotify AB
Authors: Mårten Schultzberg (Staff Data Scientist), Sebastian Ankargren (Senior Data Scientist), Mattias Frånberg (Senior Data Scientist)

> If this file and the page at the canonical URL disagree, the page is authoritative.

Risk-aware product decisions in A/B tests: how Spotify combines success and guardrail metrics into one ship or no-ship call while controlling error rates.

## TL;DR

Most experiments have multiple metrics. How do you combine them into a single ship/no-ship decision?

Metrics can be of different types: [success metrics](/glossary/success-metric) (superiority tests), [guardrail metrics](/glossary/guardrail-metric) ([non-inferiority tests](/glossary/non-inferiority-test)), and others. The decision rule we use: ship if the treatment is significantly superior on at least one success metric, and significantly non-inferior on all guardrail metrics.

Under this rule, you only need to adjust the [false positive rate](/glossary/false-positive-rate) for the number of success metrics—that's the only group where you have multiple chances to "win." You don't adjust for guardrail metrics. However, to maintain intended power, you must correct [false negative rates](/glossary/false-negative-rate) for the number of guardrail metrics. This post explains how Spotify's decision-making engine combines multiple metrics into product decisions.

**Read the full post on Spotify Engineering:** [Risk-Aware Product Decisions in A/B Tests with Multiple Metrics](https://engineering.atspotify.com/2024/03/risk-aware-product-decisions-in-a-b-tests-with-multiple-metrics)