Skip to main content

The description of which hypotheses that are going to be tested, which group comparisons that are included, and how the result should be evaluated.

alpha
number
required

The desired overall false positive rate (multiple testing will be applied automatically, so you don't need to adjust this number).

power
number
required

The desired statistical power, will be adjusted according to the decision rule.

groups
object[]
required

The list of groups/treatments that will be included.

comparisons
object
required

Description how groups will be compared to each other. Note that some analysis methods requires all groups, in those cases that will override this behaviour.

hypotheses
object[]
required

List of hypotheses that should be tested, typically these represents the change in one or more metrics.

decisionRule
object

A nested expression that decides when it is valid to "ship" the experiment. This expression will impact both how the multiple testing adjustment and power adjustment is performed, and determine the shipping recommendations.

A common example is success and guardrail metrics, for example with two success metric hypotheses S1, S2 and two guardrails hypotheses G1, G2 the decision rule could be formed as (G1 AND G2) AND (S1 OR S2). Assuming that the guardrails are non-inferiority hypotheses and the success metrics are superiority hypotheses, this means that both guardrails must be significantly non-inferior and at least one success metric must be significant.

If not set, then an OR rule will be assumed across all hypotheses. The logical operator that an hypotheses is part of, will also determine the logical operator between segments. Meaning that if an hypothesis is part of an AND expression, the rule will only evaluate to true if it is significant on all segments, and similar for an OR expression.

settings
object

Settings that control how the analysis is performed.