> ## Documentation Index
> Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Create an Analysis Plan
> Learn how to create an analysis plan for your experiment.
export const HowToSchema = ({name, description, steps = []}) => {
const schema = {
"@context": "https://schema.org",
"@type": "HowTo",
name,
description
};
if (steps.length > 0) {
schema.step = steps.map((s, i) => ({
"@type": "HowToStep",
position: i + 1,
name: typeof s === "string" ? s : s.name,
text: typeof s === "string" ? s : s.text || s.name
}));
}
return ;
};
The first step when setting up the analysis plan is to decide on the risk management parameters alpha and power, which control the overall false positive and false negative rates of the analysis. For this example, use a false positive rate of `0.05` and a statistical power of `0.8`, implying a false negative rate of `1 - 0.8 = 0.2`.
With these parameters, the analysis plan looks like:
```json theme={null}
{
"alpha": 0.05,
"power": 0.8
}
```
## Create the Groups
In this analysis, you only have two groups: a `control` group and a `treatment` group. Both have the same number of samples on average, so set both of their weights to 1.
```json theme={null}
{
"groups": [
{
"id": "control",
"weight": 1
},
{
"id": "treatment",
"weight": 1
}
]
}
```
## Set Up the Comparisons
To compare the treatment group to the control group, use a `oneVsAll` comparison specification.
```json theme={null}
{
"comparisons": {
"oneVsAll": {
"baseline": "control"
}
}
}
```
## Set Up the Hypotheses
The hypotheses consist of two metrics: a crash rate metric and a consumption metric. Use a non-inferiority hypothesis for the crash rate metric to accept a slight increase. Set the margin to 1%. To avoid waiting until the end of the experiment to learn about increased crash rates, analyze the crash rate sequentially. Use a group sequential test to analyze the data sequentially.
For the consumption metric, use a superiority hypothesis with a minimum detectable effect of 3%. Anything less than 3% is not practically meaningful in this case, so you want to design the test for this effect size. You want to analyze the consumption at the end of the experiment, so select a regular z-test.
Add a single segment entry with an empty dimensions map to signal that the test has no segmentation. Your analysis plan at this stage is:
```json theme={null}
{
"hypotheses": [
{
"id": "crashes",
"type": {
"nonInferiority": {
"preferredDirection": "DECREASE",
"nonInferiorityMargin": 0.01
}
},
"segments": [
{
"dimensions": {},
"params": {
"gstZTest": {
"maxSampleSize": 1000
}
}
}
]
},
{
"id": "consumption",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"zTest": {}
}
}
]
}
]
}
```
## Create the Decision Rule
Your decision rule is to ship the experiment if the guardrail is significantly non-inferior and the success metric is significantly superior. The decision rule is logically `crashes AND consumption`. You write it as:
```json theme={null}
{
"decisionRule": {
"items": [
{
"hypothesis": "consumption"
},
{
"hypothesis": "crashes"
}
],
"operator": "AND"
}
}
```
## Bring It All Together
Your analysis plan is now ready! The full plan is:
```json theme={null}
{
"alpha": 0.05,
"power": 0.8,
"groups": [
{
"id": "control",
"weight": 1
},
{
"id": "treatment",
"weight": 1
}
],
"comparisons": {
"oneVsAll": {
"baseline": "control"
}
},
"decisionRule": {
"items": [
{
"hypothesis": "consumption"
},
{
"hypothesis": "crashes"
}
],
"operator": "AND"
},
"hypotheses": [
{
"id": "consumption",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"zTest": {}
}
}
]
},
{
"id": "crashes",
"type": {
"nonInferiority": {
"preferredDirection": "DECREASE",
"nonInferiorityMargin": 0.01
}
},
"segments": [
{
"dimensions": {},
"params": {
"gstZTest": {
"maxSampleSize": 1000
}
}
}
]
}
]
}
```