AnalysisPlan

The description of which hypotheses that are going to be tested, which group comparisons that are included, and how the result should be evaluated.

alpha

number

required

The desired overall false positive rate (multiple testing will be applied automatically, so you don't need to adjust this number).

power

number

required

The desired statistical power, will be adjusted according to the decision rule.

groups

object[]

required

The list of groups/treatments that will be included.

Show child attributes

comparisons

object

required

Description how groups will be compared to each other. Note that some analysis methods requires all groups, in those cases that will override this behaviour.

Show child attributes

hypotheses

object[]

required

List of hypotheses that should be tested, typically these represents the change in one or more metrics.

Show child attributes

decisionRule

object

A nested expression that decides when it is valid to "ship" the experiment. This expression will impact both how the multiple testing adjustment and power adjustment is performed, and determine the shipping recommendations.

A common example is success and guardrail metrics, for example with two success metric hypotheses S1, S2 and two guardrails hypotheses G1, G2 the decision rule could be formed as (G1 AND G2) AND (S1 OR S2). Assuming that the guardrails are non-inferiority hypotheses and the success metrics are superiority hypotheses, this means that both guardrails must be significantly non-inferior and at least one success metric must be significant.

If not set, then an OR rule will be assumed across all hypotheses. The logical operator that an hypotheses is part of, will also determine the logical operator between segments. Meaning that if an hypothesis is part of an AND expression, the rule will only evaluate to true if it is significant on all segments, and similar for an OR expression.

Show child attributes

settings

object

Settings that control how the analysis is performed.

Show child attributes

AnalysisData

⌘I

{
  "alpha": 123,
  "power": 123,
  "groups": [
    {
      "id": "<string>",
      "weight": 123
    }
  ],
  "comparisons": {
    "oneVsAll": {
      "baseline": "<string>"
    },
    "allVsAll": {},
    "custom": {
      "pairs": [
        {
          "baseline": "<string>",
          "compared": "<string>"
        }
      ]
    }
  },
  "hypotheses": [
    {
      "id": "<string>",
      "type": {
        "superiority": {
          "preferredDirection": {},
          "minimumDetectableEffect": 123
        },
        "nonInferiority": {
          "preferredDirection": {},
          "nonInferiorityMargin": 123
        },
        "jointEquivalence": {
          "minimumDetectableDeviation": 123
        }
      },
      "segments": [
        {
          "params": {
            "chi2": {
              "expectedFrequency": [
                {
                  "group": "<string>",
                  "frequency": 123
                }
              ]
            },
            "zTest": {},
            "gstZTest": {
              "maxSampleSize": 123
            },
            "ratio": {},
            "tunncDesign": {
              "optimalSampleSize": 123,
              "weight": [
                {}
              ]
            },
            "gstRatio": {
              "maxSampleSize": 123
            },
            "asympCs": {
              "optimalSampleSize": 123
            },
            "asympCsRatio": {
              "optimalSampleSize": 123
            }
          },
          "dimensions": [
            {}
          ]
        }
      ],
      "dataType": {}
    }
  ],
  "decisionRule": {
    "items": [
      {
        "hypothesis": "<string>",
        "rule": {}
      }
    ],
    "operator": {}
  },
  "settings": {
    "disableSegmentAlphaAdjustment": true
  }
}

Get Started

Feature Flags

Metrics

Stats

Experiments

Surfaces

IAM

Connectors