Skip to main content

Documentation Index

Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt

Use this file to discover all available pages before exploring further.

When analyzing experiments with multiple metrics, adjustments control the overall false positive rate. The Stats API uses decision rules to determine how to adjust alpha and power levels for each metric.

​
Decision Rules

The analysis plan contains a decisionRule object that specifies how metrics combine to form an overall shipping decision. The decision rule uses AND and OR operators between hypotheses.

​
Structure

{
  "decisionRule": {
    "items": [
      { "hypothesis": "success-metric" },
      { "hypothesis": "guardrail-metric" }
    ],
    "operator": "AND"
  }
}
This example encodes: “Ship if success-metric is significant AND guardrail-metric is significant.”

​
How Adjustments Work

The multiple comparison adjustments control the false positive rate of the overall decision rule using:
  • Union-intersection testing for OR conditions (at least one must be significant)
  • Intersection-union testing for AND conditions (all must be significant)
The adjustment depends on each metric’s role within the decision rule. Metrics joined by OR receive stricter alpha adjustments, while metrics joined by AND receive power adjustments.

​
Default Behavior

If you don’t specify a decision rule, the API applies a Bonferroni multiple testing correction. This divides the alpha level by the number of hypotheses. 
{
  "alpha": 0.05,
  "power": 0.8,
  "hypotheses": [
    { "id": "metric-1", "..." },
    { "id": "metric-2", "..." },
    { "id": "metric-3", "..." }
  ]
}
With three hypotheses and alpha = 0.05, each hypothesis uses alpha = 0.0167.

​
Example: Success and Guardrail Metrics

A common pattern tests for improvement in a success metric while ensuring guardrails pass: 
{
  "alpha": 0.05,
  "power": 0.8,
  "hypotheses": [
    {
      "id": "consumption",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "zTest": {}
          }
        }
      ]
    },
    {
      "id": "crashes",
      "type": {
        "nonInferiority": {
          "preferredDirection": "DECREASE",
          "nonInferiorityMargin": 0.01
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "gstZTest": {
              "maxSampleSize": 1000
            }
          }
        }
      ]
    }
  ],
  "decisionRule": {
    "items": [
      { "hypothesis": "consumption" },
      { "hypothesis": "crashes" }
    ],
    "operator": "AND"
  }
}
This tests: “Ship if consumption improves AND crashes don’t degrade beyond the margin.”

Multiple Comparisons

Learn about multiple comparison concepts

Create Analysis Plan

Set up your analysis plan

Statistical Tests

Understand test types

Analyze Results

Understand decision rules