> ## Documentation Index
> Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Multiple Comparisons Adjustment

> Use decision rules in the Stats API to control false positive rates across multiple metrics.

export const HowToSchema = ({name, description, steps = []}) => {
  const schema = {
    "@context": "https://schema.org",
    "@type": "HowTo",
    name,
    description
  };
  if (steps.length > 0) {
    schema.step = steps.map((s, i) => ({
      "@type": "HowToStep",
      position: i + 1,
      name: typeof s === "string" ? s : s.name,
      text: typeof s === "string" ? s : s.text || s.name
    }));
  }
  return <script type="application/ld+json" dangerouslySetInnerHTML={{
    __html: JSON.stringify(schema)
  }} />;
};

<HowToSchema
  name="Configure Multiple Comparisons Adjustment"
  description="Use decision rules in the Stats API to control false positive rates across multiple metrics."
  steps={[
{
  name: "Decision Rules",
  text: 'The analysis plan contains a decisionRule object that specifies how metrics combine to form an overall shipping decision. The decision rule uses AND and OR operators between hypotheses. This example encodes: "Ship if success-metric is significant AND guardrail-metric is significant." The multiple comparison adjustments control the false positive rate of the overall decision rule using:',
},
{
  name: "Default Behavior",
  text: "If you don't specify a decision rule, the API applies a Bonferroni multiple testing correction. This divides the alpha level by the number of hypotheses. With three hypotheses and alpha = 0.05, each hypothesis uses alpha = 0.0167.",
},
{
  name: "Example: Success and Guardrail Metrics",
  text: 'A common pattern tests for improvement in a success metric while ensuring guardrails pass: This tests: "Ship if consumption improves AND crashes don\'t degrade beyond the margin."',
},
]}
/>

When analyzing experiments with multiple metrics, adjustments control the overall false positive rate. The Stats API uses decision rules to determine how to adjust alpha and power levels for each metric.

## Decision Rules

The analysis plan contains a `decisionRule` object that specifies how metrics combine to form an overall shipping decision. The decision rule uses `AND` and `OR` operators between hypotheses.

### Structure

```json theme={null}
{
  "decisionRule": {
    "items": [
      { "hypothesis": "success-metric" },
      { "hypothesis": "guardrail-metric" }
    ],
    "operator": "AND"
  }
}
```

This example encodes: "Ship if success-metric is significant AND guardrail-metric is significant."

### How Adjustments Work

The multiple comparison adjustments control the false positive rate of the overall decision rule using:

* **Union-intersection testing** for OR conditions (at least one must be significant)
* **Intersection-union testing** for AND conditions (all must be significant)

The adjustment depends on each metric's role within the decision rule. Metrics joined by OR receive stricter alpha adjustments, while metrics joined by AND receive power adjustments.

## Default Behavior

If you don't specify a decision rule, the API applies a Bonferroni multiple testing correction. This divides the alpha level by the number of hypotheses.

```json theme={null}
{
  "alpha": 0.05,
  "power": 0.8,
  "hypotheses": [
    { "id": "metric-1", "..." },
    { "id": "metric-2", "..." },
    { "id": "metric-3", "..." }
  ]
}
```

With three hypotheses and alpha = 0.05, each hypothesis uses alpha = 0.0167.

## Example: Success and Guardrail Metrics

A common pattern tests for improvement in a success metric while ensuring guardrails pass:

```json theme={null}
{
  "alpha": 0.05,
  "power": 0.8,
  "hypotheses": [
    {
      "id": "consumption",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "zTest": {}
          }
        }
      ]
    },
    {
      "id": "crashes",
      "type": {
        "nonInferiority": {
          "preferredDirection": "DECREASE",
          "nonInferiorityMargin": 0.01
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "gstZTest": {
              "maxSampleSize": 1000
            }
          }
        }
      ]
    }
  ],
  "decisionRule": {
    "items": [
      { "hypothesis": "consumption" },
      { "hypothesis": "crashes" }
    ],
    "operator": "AND"
  }
}
```

This tests: "Ship if consumption improves AND crashes don't degrade beyond the margin."

## Related Resources

<CardGroup cols={2}>
  <Card title="Multiple Comparisons" href="/docs/experiments/stats/adjustment-multiple-comparisons">
    Learn about multiple comparison concepts
  </Card>

  <Card title="Create Analysis Plan" href="/docs/api/how-to-guides/stats/create-analysis-plan">
    Set up your analysis plan
  </Card>

  <Card title="Statistical Tests" href="/docs/experiments/stats/stat-tests">
    Understand test types
  </Card>

  <Card title="Analyze Results" href="/docs/experiments/analyze-results">
    Understand decision rules
  </Card>
</CardGroup>