> ## Documentation Index > Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt > Use this file to discover all available pages before exploring further. # Run Analysis > Runs an analysis that tests multiple hypotheses and evaluates their results by an optional decision rule. ## OpenAPI ````yaml /api/stats/api/analysis.openapi.json post /v2/stats:runAnalysis openapi: 3.1.0 info: title: Confidence Analysis API version: 1.0.0 description: API documentation for analysis servers: [] security: [] paths: /v2/stats:runAnalysis: post: summary: Run Analysis description: >- Runs an analysis that tests multiple hypotheses and evaluates their results by an optional decision rule. operationId: Analysis.run parameters: [] requestBody: content: application/json: schema: type: object properties: plan: $ref: '#/components/schemas/confidence.stats.v2.AnalysisPlan' description: >- [AnalysisPlan](/api-reference/schemas/analysisplan): The description of which hypotheses that are going to be tested, which group comparisons that are included, and how the result should be evaluated. data: $ref: '#/components/schemas/confidence.stats.v2.AnalysisData' description: >- [AnalysisData](/api-reference/schemas/analysisdata): The data for each hypothesis. required: - plan - data required: true responses: '200': description: OK content: application/json: schema: type: object properties: hypotheses: type: array items: type: object properties: id: type: string description: The id of the hypothesis. segments: type: array items: type: object properties: dimensions: type: array items: type: object additionalProperties: type: string description: >- The set of dimensions and their values that represent the segment. comparisons: type: array items: type: object properties: groups: type: object properties: single: type: string description: >- A single group that is compared to an expected value. pair: type: object properties: baseline: type: string description: The baseline group. compared: type: string description: >- The group being compared to the baseline. required: - baseline - compared description: Two groups compared to each other. multiple: type: object properties: groups: type: array items: type: string description: The list of groups being compared. required: - groups description: Multiple groups compared to each other. description: The groups part of the comparison. result: type: object properties: chi2: type: object properties: frequency: type: array items: type: object properties: group: type: string description: The user-defined id of the group. value: type: number description: >- Estimate value of the frequency in the group. expected: type: number description: The expected frequency in the group. description: Estimated frequency of each group. chi2Statistic: type: number description: The χ² test statistic. pValue: type: number description: The p-value of the test statistic. description: Results of the χ² test. zTest: type: object properties: baselineMeanAbs: type: number description: >- The mean of the baseline group on the original scale. comparedMeanAbs: type: number description: >- The mean of the compared group on the original scale. baselineMeanRel: type: number description: >- The mean of the baseline group relative to itself. comparedMeanRel: type: number description: >- The mean of the compared group relative to the baseline. differenceAbs: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: >- Estimated difference on the original scale. differenceRel: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: >- Estimated difference relative to the baseline. zStatistic: type: number description: Z-test statistic. pValue: type: number description: P-value of the z-test. varianceReductionRate: type: number description: >- Variance reduction rate, set if variance reduction was used. unadjustedBaselineMeanAbs: type: number description: >- Baseline ratio not adjusted for variance reduction. unadjustedComparedMeanAbs: type: number description: >- Compared ratio not adjusted for variance reduction. baselineSampleSize: type: number description: Sample size for the baseline group comparedSampleSize: type: number description: Sample size for the compared group description: Results of the z-test. gstZTest: type: object properties: baselineMeanAbs: type: array items: type: number description: >- The mean of the baseline group on the original scale. comparedMeanAbs: type: array items: type: number description: >- The mean of the compared group on the original scale. baselineMeanRel: type: array items: type: number description: >- The mean of the baseline group relative to itself. comparedMeanRel: type: array items: type: number description: >- The mean of the compared group relative to the baseline. differenceAbs: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated sequential difference on the original scale. differenceRel: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated sequential difference relative to the baseline. zStatistic: type: array items: type: number description: >- Z-statistics for each analysis time of the sequential test. varianceReductionRate: type: array items: type: number description: >- Variance reduction rate, set if variance reduction was used. unadjustedBaselineMeanAbs: type: array items: type: number description: >- Baseline mean not adjusted for variance reduction. unadjustedComparedMeanAbs: type: array items: type: number description: >- Compared mean not adjusted for variance reduction. baselineSampleSize: type: array items: type: number description: Sample size for the baseline group comparedSampleSize: type: array items: type: number description: Sample size for the compared group description: Results of the sequential z-test. ratio: type: object properties: baselineRatioAbs: type: number description: >- The ratio of the baseline group on the original scale. comparedRatioAbs: type: number description: >- The ratio of the compared group on the original scale. baselineRatioRel: type: number description: >- The ratio of the baseline group relative to itself. comparedRatioRel: type: number description: >- The ratio of the compared group relative to the baseline. ratioDifferenceAbs: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: >- Estimated ratio difference on the original scale. ratioDifferenceRel: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: >- Estimated ratio difference relative to the baseline. zStatistic: type: number description: >- Z-statistic for the difference in ratios. pValue: type: number description: P-value of the z-statistic. varianceReductionRate: type: number description: Variance reduction rate unadjustedBaselineRatioAbs: type: number description: >- Baseline ratio not adjusted for variance reduction. unadjustedComparedRatioAbs: type: number description: >- Compared ratio not adjusted for variance reduction. unadjustedBaselineNumerator: type: number description: >- Baseline numerator not adjusted for variance reduction. unadjustedComparedNumerator: type: number description: >- Compared numerator not adjusted for variance reduction. unadjustedBaselineDenominator: type: number description: >- Baseline denominator not adjusted for variance reduction. unadjustedComparedDenominator: type: number description: >- Compared denominator not adjusted for variance reduction. baselineSampleSize: type: number description: Sample size for the baseline group comparedSampleSize: type: number description: Sample size for the compared group description: Results of the ratio test. tunncDesign: type: object properties: baselineMeanAbs: type: array items: type: number description: >- The mean of the baseline group on the original scale. comparedMeanAbs: type: array items: type: number description: >- The mean of the compared group on the original scale. baselineMeanRel: type: array items: type: number description: >- The mean of the baseline group relative to itself. comparedMeanRel: type: array items: type: number description: >- The mean of the compared group relative to the baseline. differenceAbs: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated sequential difference on the original scale. differenceRel: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated sequential difference relative to the baseline. varianceReductionRate: type: array items: type: number description: >- Variance reduction rate, set if variance reduction was used. description: >- Results of the non-parametric sequential test. gstRatio: type: object properties: baselineRatioAbs: type: array items: type: number description: >- The ratio of the baseline group on the original scale. comparedRatioAbs: type: array items: type: number description: >- The ratio of the compared group on the original scale. baselineRatioRel: type: array items: type: number description: >- The ratio of the baseline group relative to itself. comparedRatioRel: type: array items: type: number description: >- The ratio of the compared group relative to the baseline. ratioDifferenceAbs: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated ratio difference on the original scale. ratioDifferenceRel: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated ratio difference relative to the baseline. zStatistic: type: array items: type: number description: >- Z-statistic for the difference in ratios. varianceReductionRate: type: array items: type: number description: Variance reduction rates unadjustedBaselineRatioAbs: type: array items: type: number description: >- Baseline ratio not adjusted for variance reduction. unadjustedComparedRatioAbs: type: array items: type: number description: >- Compared ratio not adjusted for variance reduction. unadjustedBaselineNumerator: type: array items: type: number description: >- Baseline numerator not adjusted for variance reduction. unadjustedComparedNumerator: type: array items: type: number description: >- Compared numerator not adjusted for variance reduction. unadjustedBaselineDenominator: type: array items: type: number description: >- Baseline denominator not adjusted for variance reduction. unadjustedComparedDenominator: type: array items: type: number description: >- Compared denominator not adjusted for variance reduction. baselineSampleSize: type: array items: type: number description: Sample size for the baseline group comparedSampleSize: type: array items: type: number description: Sample size for the compared group description: Results of the ratio test. asympCs: type: object properties: baselineMeanAbs: type: array items: type: number description: >- The mean of the baseline group on the original scale. comparedMeanAbs: type: array items: type: number description: >- The mean of the compared group on the original scale. baselineMeanRel: type: array items: type: number description: >- The mean of the baseline group relative to itself. comparedMeanRel: type: array items: type: number description: >- The mean of the compared group relative to the baseline. differenceAbs: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated sequential difference on the original scale. differenceRel: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated sequential difference relative to the baseline. varianceReductionRate: type: array items: type: number description: >- Variance reduction rate, set if variance reduction was used. unadjustedBaselineMeanAbs: type: array items: type: number description: >- Baseline mean not adjusted for variance reduction. unadjustedComparedMeanAbs: type: array items: type: number description: >- Compared mean not adjusted for variance reduction. baselineSampleSize: type: array items: type: number description: Sample size for the baseline group comparedSampleSize: type: array items: type: number description: Sample size for the compared group description: >- Results of the non-parametric sequential test. asympCsRatio: type: object properties: baselineRatioAbs: type: array items: type: number description: >- The ratio of the baseline group on the original scale. comparedRatioAbs: type: array items: type: number description: >- The ratio of the compared group on the original scale. baselineRatioRel: type: array items: type: number description: >- The ratio of the baseline group relative to itself. comparedRatioRel: type: array items: type: number description: >- The ratio of the compared group relative to the baseline. ratioDifferenceAbs: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated ratio difference on the original scale. ratioDifferenceRel: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time as passed by the user. estimate: type: object properties: estimate: type: number description: Value of the estimated parameter. stdErr: type: number description: >- Standard error of the estimated parameter if it exists, otherwise nan. lower: type: number description: >- Lower part of the confidence interval of the estimated parameter. upper: type: number description: >- Upper part of the confidence interval of the estimated parameter. powered: type: number description: >- The currently powered effect, only set if power parameters were passed, otherwise nan. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the parameter is significant according to the given hypothesis, if there was no hypothesis for this parameter it is not set. description: The estimated parameter. description: >- Estimated ratio difference relative to the baseline. zStatistic: type: array items: type: number description: >- Z-statistic for the difference in ratios. varianceReductionRate: type: array items: type: number description: Variance reduction rates unadjustedBaselineRatioAbs: type: array items: type: number description: >- Baseline ratio not adjusted for variance reduction. unadjustedComparedRatioAbs: type: array items: type: number description: >- Compared ratio not adjusted for variance reduction. unadjustedBaselineNumerator: type: array items: type: number description: >- Baseline numerator not adjusted for variance reduction. unadjustedComparedNumerator: type: array items: type: number description: >- Compared numerator not adjusted for variance reduction. unadjustedBaselineDenominator: type: array items: type: number description: >- Baseline denominator not adjusted for variance reduction. unadjustedComparedDenominator: type: array items: type: number description: >- Compared denominator not adjusted for variance reduction. baselineSampleSize: type: array items: type: number description: Sample size for the baseline group comparedSampleSize: type: array items: type: number description: Sample size for the compared group description: >- Results of the non-parametric sequential ratio test. description: The result of the analysis. currentSampleSize: type: integer description: Current sample size. requiredSampleSize: type: integer description: >- Estimate of the sample size needed to reach desired power, if it was possible to estimate, otherwise zero. isSignificant: type: object properties: value: type: boolean description: The bool value. description: >- Determines whether the result was statistically significant, not set if the analysis could not be completed. comparisonId: type: string description: >- Identifier for the comparison, used to find the corresponding annotation. usedDataType: type: string enum: - DATA_TYPE_UNSPECIFIED - DATA_TYPE_CONTINUOUS - DATA_TYPE_BINARY - DATA_TYPE_COUNT description: >- The data type actually used in the analysis. groupStats: type: array items: type: object properties: id: type: string description: The id of the group. sampleSize: type: integer description: The number of observations in the group. description: Statistics for each group. description: List of results for each group comparison. labels: type: array items: type: object additionalProperties: type: string description: >- Set of labels passed on from the analysis request. description: >- Result for each segment, if no segmentation is used, this list will contain a single result with an empty segment. adjustedAlpha: type: number description: >- The alpha (false positive rate) used for the analysis, after adjustment. For one-sided tests, this is the alpha used in the selected tail. adjustedBeta: type: number description: >- The beta (1 - power) used for the analysis, after adjustment. description: The analysis results for each hypothesis. recommendations: type: array items: type: object properties: groups: type: object properties: single: type: string description: >- A single group that is compared to an expected value. pair: type: object properties: baseline: type: string description: The baseline group. compared: type: string description: The group being compared to the baseline. required: - baseline - compared description: Two groups compared to each other. multiple: type: object properties: groups: type: array items: type: string description: The list of groups being compared. required: - groups description: Multiple groups compared to each other. description: The groups that were compared. shipmentRecommendation: type: string enum: - SHIPMENT_RECOMMENDATION_UNSPECIFIED - SHIP - MAYBE_SHIP - STOP description: >- The recommendation for the specific comparison according to the given decision rule. description: >- Overall recommendations for each comparison according to specified decision rule. annotations: type: array items: type: object properties: context: type: array items: type: object properties: key: type: string enum: - CONTEXT_UNSPECIFIED - CONTEXT_HYPOTHESIS - CONTEXT_SEGMENT - CONTEXT_GROUP - CONTEXT_COMPARISON description: Specific context location, like a hypothesis. value: type: string description: The id. description: >- The context that the annotation was surfaced in, e.g. which group, hypothesis or segment it occurred in. info: type: string description: General information, not treated as an error. warning: type: string description: Warning, something might be wrong. error: type: object properties: details: type: string description: Error message. category: type: string enum: - ERROR_CATEGORY_UNSPECIFIED - ERROR_CATEGORY_GENERAL - ERROR_CATEGORY_DATA_VALIDATION_ERROR - ERROR_CATEGORY_ANALYSIS_ERROR - ERROR_CATEGORY_NO_DATA description: General category of the error. description: Error, something has gone wrong. description: List of issues discovered during the analysis. statsCacheEntry: type: string description: Cache entry used, empty if not cached. security: - admin: [] servers: - url: https://stats.eu.confidence.dev - url: https://stats.us.confidence.dev - url: https://stats.confidence.dev components: schemas: confidence.stats.v2.AnalysisPlan: title: AnalysisPlan type: object description: |- The description of which hypotheses that are going to be tested, which group comparisons that are included, and how the result should be evaluated. properties: alpha: type: number description: >- The desired overall false positive rate (multiple testing will be applied automatically, so you don't need to adjust this number). power: type: number description: >- The desired statistical power, will be adjusted according to the decision rule. groups: type: array items: type: object properties: id: type: string description: User-defined identifier of the group. weight: type: integer description: >- The weight of the group, i.e., the number of samples allocated to group i will approximately be `weight_i / sum(weight_i)`. required: - id - weight description: The list of groups/treatments that will be included. comparisons: type: object properties: oneVsAll: type: object properties: baseline: type: string description: The group that will act as the baseline. required: - baseline description: Compare all groups against one. allVsAll: type: object description: Compare all groups against each other. custom: type: object properties: pairs: type: array items: type: object properties: baseline: type: string description: The group that will act as the baseline. compared: type: string description: The group that will be compared to the baseline. required: - baseline - compared description: List of groups that will be compared. required: - pairs description: Compare groups according to the custom list of pairs. description: >- Description how groups will be compared to each other. Note that some analysis methods requires all groups, in those cases that will override this behaviour. hypotheses: type: array items: type: object properties: id: type: string description: >- A used-defined id of the hypothesis that will be present in the result. type: type: object properties: superiority: type: object properties: preferredDirection: type: string enum: - DIRECTION_UNSPECIFIED - INCREASE - DECREASE - ANY description: >- The direction in which you want to declare superiority. minimumDetectableEffect: type: number description: >- The minimum effect that can be detected with the given power level. Typically the size of the effect that is of practical significance. The scale of this is relative to the segment baseline mean and requires the baseline to be set to be able to do power analysis. required: - preferredDirection description: >- A superiority hypothesis tests whether an estimate in one group is different, greater or lesser than an estimate of another group. nonInferiority: type: object properties: preferredDirection: type: string enum: - DIRECTION_UNSPECIFIED - INCREASE - DECREASE description: >- The direction in which you allow increases larger than the margin of error. nonInferiorityMargin: type: number description: >- The margin of error when declaring non-inferiority. The scale of this is relative to the segment baseline mean and requires the baseline to be set to be able to do power analysis. required: - preferredDirection - nonInferiorityMargin description: >- A non-inferiority hypothesis tests whether an estimate in one group is greater or lesser, with some margin of error, than an estimate of another group. jointEquivalence: type: object properties: minimumDetectableDeviation: type: number description: >- The minimum detectable deviation in all of the groups, i.e., if you have three groups with a minimum detectable deviation of 0.1 that means that if only one group differs in 0.1 it should be detectable. required: - minimumDetectableDeviation description: >- A joint equivalence hypothesis tests whether multiple estimates are the same or equal to some specified value. description: >- Determines whether you are testing a superiority or non-inferiority test. dataType: type: string enum: - DATA_TYPE_UNSPECIFIED - DATA_TYPE_CONTINUOUS - DATA_TYPE_BINARY - DATA_TYPE_COUNT description: >- The type of the underlying data, some methods might be able to take better advantage of the data if this is set, by default it is assumed that the data is continuous. segments: type: array items: type: object properties: dimensions: type: array items: type: object additionalProperties: type: string description: >- The set of dimensions and their values that represent the segment. params: type: object properties: chi2: type: object properties: expectedFrequency: type: array items: type: object properties: group: type: string description: The user-defined id of the group. frequency: type: number description: >- The expected frequency compared to the total count of this group. required: - group - frequency description: The expected frequency of each group. required: - expectedFrequency description: Analysis parameters for the χ² test. zTest: type: object description: Analysis parameters for the z-test. gstZTest: type: object properties: maxSampleSize: type: integer description: >- An estimate of the maximum sample size for the segment. required: - maxSampleSize description: Analysis parameters for the sequential z-test. ratio: type: object description: Analysis parameters for the ratio test. tunncDesign: type: object properties: optimalSampleSize: type: number description: >- Parameter that determines at which sample size where the bounds for the confidence intervals will be as tight as possible. It must be a positive number. The method is not too sensitive to this number, but err on underestimating it. If not set, a default value of 1 will be used. weight: type: array items: type: object additionalProperties: type: number description: >- Optionally override the weight passed the non-parametric test, if not set the group weights will be used. description: >- Analysis parameters for the non-parametric sequential test. gstRatio: type: object properties: maxSampleSize: type: integer description: >- An estimate of the maximum sample size for the segment. required: - maxSampleSize description: Analysis parameters for the sequential ratio test. asympCs: type: object properties: optimalSampleSize: type: number description: >- Parameter that determines at which sample size where the bounds for the confidence intervals will be as tight as possible. It must be a positive number. The method is not too sensitive to this number, but err on underestimating it. If not set, a default value of 10 will be used. description: >- Analysis parameters for the non-parametric sequential test. asympCsRatio: type: object properties: optimalSampleSize: type: number description: >- Parameter that determines at which sample size where the bounds for the confidence intervals will be as tight as possible. It must be a positive number. The method is not too sensitive to this number, but err on underestimating it. If not set, a default value of 10 will be used. description: >- Analysis parameters for the non-parametric sequential ratio test. description: The method that will be used to test the hypothesis, required: - params description: >- List of segments to test. If you don't need segmentation, you can specify the overall segment with an empty map. required: - id - type - segments description: >- List of hypotheses that should be tested, typically these represents the change in one or more metrics. decisionRule: type: object properties: items: type: array items: type: object properties: hypothesis: type: string description: The id of a hypothesis. rule: type: object description: A nested decision rule. description: >- A possibly nested boolean expression, where each term is either the id of an hypothesis or a nested boolean expression. operator: type: string enum: - LOGICAL_OPERATOR_UNSPECIFIED - AND - OR description: A boolean operator AND or OR that occurs between the items. required: - items - operator description: >- A nested expression that decides when it is valid to "ship" the experiment. This expression will impact both how the multiple testing adjustment and power adjustment is performed, and determine the shipping recommendations. A common example is success and guardrail metrics, for example with two success metric hypotheses S1, S2 and two guardrails hypotheses G1, G2 the decision rule could be formed as (G1 AND G2) AND (S1 OR S2). Assuming that the guardrails are non-inferiority hypotheses and the success metrics are superiority hypotheses, this means that both guardrails must be significantly non-inferior and at least one success metric must be significant. If not set, then an OR rule will be assumed across all hypotheses. The logical operator that an hypotheses is part of, will also determine the logical operator between segments. Meaning that if an hypothesis is part of an AND expression, the rule will only evaluate to true if it is significant on all segments, and similar for an OR expression. settings: type: object properties: disableSegmentAlphaAdjustment: type: boolean description: >- If true, disables the Bonferroni correction for the number of segments. By default, alpha is adjusted for the total number of tests including all segments. Set to true if you want to treat segments as independent analyses without multiple testing correction across them. description: Settings that control how the analysis is performed. required: - alpha - power - groups - comparisons - hypotheses confidence.stats.v2.AnalysisData: title: AnalysisData type: object description: Data used for the analysis. properties: hypotheses: type: array items: type: object properties: id: type: string description: The id of the hypothesis. segments: type: array items: type: object properties: dimensions: type: array items: type: object additionalProperties: type: string description: >- The set of dimensions and their values that represent the segment. groups: type: array items: type: object properties: group: type: string description: The id of the group. data: type: object properties: chi2: type: object properties: summary: type: object properties: count: type: integer description: The count of the group. required: - count description: Run the analysis from summary data. description: Data for the χ² test. zTest: type: object properties: summary: type: object properties: mean: type: number description: Estimated mean of the group. variance: type: number description: Estimated variance of the group. count: type: integer description: Number of samples in the group. adjustment: type: object properties: mean: type: number description: Mean of the covariate. variance: type: number description: Variance of the covariate. covariance: type: number description: >- Covariance of the covariate with the variable used for the mean. completeMean: type: number description: >- Mean of the outcome for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - mean - variance - covariance description: >- Optional summary data for a covariate adjustment. required: - mean - variance - count description: Run the analysis from summary data. description: Data for the z-test. gstZTest: type: object properties: summary: type: object properties: data: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time, must be sortable. value: type: object properties: mean: type: number description: Estimated mean of the group. variance: type: number description: Estimated variance of the group. count: type: integer description: Number of samples in the group. adjustment: type: object properties: mean: type: number description: Mean of the covariate. variance: type: number description: Variance of the covariate. covariance: type: number description: >- Covariance of the covariate with the variable used for the mean. completeMean: type: number description: >- Mean of the outcome for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - mean - variance - covariance description: >- Optional summary data for a covariate adjustment. required: - mean - variance - count description: Value of the summary data. required: - timeLabel - value description: >- Summary data for each time point of the sequential analysis, note that the count must be increasing at each time point. required: - data description: Run the analysis from summary data. description: Data for the sequential z-test. ratio: type: object properties: summary: type: object properties: numeratorMean: type: number description: Mean of the numerator. numeratorVariance: type: number description: Variance of the numerator. denominatorMean: type: number description: Mean of the denominator. denominatorVariance: type: number description: Variance of the denominator. covariance: type: number description: >- Covariance of the numerator and denominator. count: type: integer description: Total number of samples. adjustment: type: object properties: numeratorMean: type: number description: Mean of the numerator. numeratorVariance: type: number description: Variance of the numerator. denominatorMean: type: number description: Mean of the denominator. denominatorVariance: type: number description: Variance of the denominator. covariance: type: number description: >- Covariance of the numerator and denominator. preNumeratorPostNumeratorCovariance: type: number description: >- Covariance between the numerator pre and post exposure preNumeratorPostDenominatorCovariance: type: number description: >- Covariance between the numerator pre exposure and denominator post exposure preDenominatorPostNumeratorCovariance: type: number description: >- Covariance between the denominator pre exposure and numerator post exposure preDenominatorPostDenominatorCovariance: type: number description: >- Covariance between the denominator pre exposure and denominator post exposure completeNumeratorMean: type: number description: >- Mean of the outcome numerator for samples that have non-missing values for both the outcome and covariate. completeDenominatorMean: type: number description: >- Mean of the outcome denominator for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - numeratorMean - numeratorVariance - denominatorMean - denominatorVariance - covariance - preNumeratorPostNumeratorCovariance - preNumeratorPostDenominatorCovariance - preDenominatorPostNumeratorCovariance - preDenominatorPostDenominatorCovariance description: >- Optional summary data for a covariate adjustment. required: - numeratorMean - numeratorVariance - denominatorMean - denominatorVariance - covariance - count description: Run the analysis from summary data. description: Data for the ratio test. tunncDesign: type: object properties: summary: type: object properties: data: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time, must be sortable. value: type: object properties: mean: type: number description: Estimated mean of the group. variance: type: number description: Estimated variance of the group. count: type: integer description: Number of samples in the group. adjustment: type: object properties: mean: type: number description: Mean of the covariate. variance: type: number description: Variance of the covariate. covariance: type: number description: >- Covariance of the covariate with the variable used for the mean. completeMean: type: number description: >- Mean of the outcome for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - mean - variance - covariance description: >- Optional summary data for a covariate adjustment. required: - mean - variance - count description: Value of the summary data. required: - timeLabel - value description: >- Summary data for each time point of the sequential analysis, note that the count must be increasing at each time point. required: - data description: Run the analysis from summary data. description: Data for the non-parametric sequential test. gstRatio: type: object properties: summary: type: object properties: data: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time, must be sortable. value: type: object properties: numeratorMean: type: number description: Mean of the numerator. numeratorVariance: type: number description: Variance of the numerator. denominatorMean: type: number description: Mean of the denominator. denominatorVariance: type: number description: Variance of the denominator. covariance: type: number description: >- Covariance of the numerator and denominator. count: type: integer description: Total number of samples. adjustment: type: object properties: numeratorMean: type: number description: Mean of the numerator. numeratorVariance: type: number description: Variance of the numerator. denominatorMean: type: number description: Mean of the denominator. denominatorVariance: type: number description: Variance of the denominator. covariance: type: number description: >- Covariance of the numerator and denominator. preNumeratorPostNumeratorCovariance: type: number description: >- Covariance between the numerator pre and post exposure preNumeratorPostDenominatorCovariance: type: number description: >- Covariance between the numerator pre exposure and denominator post exposure preDenominatorPostNumeratorCovariance: type: number description: >- Covariance between the denominator pre exposure and numerator post exposure preDenominatorPostDenominatorCovariance: type: number description: >- Covariance between the denominator pre exposure and denominator post exposure completeNumeratorMean: type: number description: >- Mean of the outcome numerator for samples that have non-missing values for both the outcome and covariate. completeDenominatorMean: type: number description: >- Mean of the outcome denominator for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - numeratorMean - numeratorVariance - denominatorMean - denominatorVariance - covariance - preNumeratorPostNumeratorCovariance - preNumeratorPostDenominatorCovariance - preDenominatorPostNumeratorCovariance - preDenominatorPostDenominatorCovariance description: >- Optional summary data for a covariate adjustment. required: - numeratorMean - numeratorVariance - denominatorMean - denominatorVariance - covariance - count description: Value of the summary data. required: - timeLabel - value description: >- Summary data for each time point of the sequential analysis, note that the count must be increasing at each time point. required: - data description: >- Run the analysis from sequential summary data. description: Data for the sequential ratio test. asympCs: type: object properties: summary: type: object properties: data: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time, must be sortable. value: type: object properties: mean: type: number description: Estimated mean of the group. variance: type: number description: Estimated variance of the group. count: type: integer description: Number of samples in the group. adjustment: type: object properties: mean: type: number description: Mean of the covariate. variance: type: number description: Variance of the covariate. covariance: type: number description: >- Covariance of the covariate with the variable used for the mean. completeMean: type: number description: >- Mean of the outcome for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - mean - variance - covariance description: >- Optional summary data for a covariate adjustment. required: - mean - variance - count description: Value of the summary data. required: - timeLabel - value description: >- Summary data for each time point of the sequential analysis, note that the count must be increasing at each time point. required: - data description: Run the analysis from summary data. description: Data for the non-parametric sequential test. asympCsRatio: type: object properties: summary: type: object properties: data: type: array items: type: object properties: timeLabel: type: string description: >- General string label for time, must be sortable. value: type: object properties: numeratorMean: type: number description: Mean of the numerator. numeratorVariance: type: number description: Variance of the numerator. denominatorMean: type: number description: Mean of the denominator. denominatorVariance: type: number description: Variance of the denominator. covariance: type: number description: >- Covariance of the numerator and denominator. count: type: integer description: Total number of samples. adjustment: type: object properties: numeratorMean: type: number description: Mean of the numerator. numeratorVariance: type: number description: Variance of the numerator. denominatorMean: type: number description: Mean of the denominator. denominatorVariance: type: number description: Variance of the denominator. covariance: type: number description: >- Covariance of the numerator and denominator. preNumeratorPostNumeratorCovariance: type: number description: >- Covariance between the numerator pre and post exposure preNumeratorPostDenominatorCovariance: type: number description: >- Covariance between the numerator pre exposure and denominator post exposure preDenominatorPostNumeratorCovariance: type: number description: >- Covariance between the denominator pre exposure and numerator post exposure preDenominatorPostDenominatorCovariance: type: number description: >- Covariance between the denominator pre exposure and denominator post exposure completeNumeratorMean: type: number description: >- Mean of the outcome numerator for samples that have non-missing values for both the outcome and covariate. completeDenominatorMean: type: number description: >- Mean of the outcome denominator for samples that have non-missing values for both the outcome and covariate. completeCount: type: integer description: >- Number of samples in the group that have non-missing values for both the outcome and covariate. required: - numeratorMean - numeratorVariance - denominatorMean - denominatorVariance - covariance - preNumeratorPostNumeratorCovariance - preNumeratorPostDenominatorCovariance - preDenominatorPostNumeratorCovariance - preDenominatorPostDenominatorCovariance description: >- Optional summary data for a covariate adjustment. required: - numeratorMean - numeratorVariance - denominatorMean - denominatorVariance - covariance - count description: Value of the summary data. required: - timeLabel - value description: >- Summary data for each time point of the sequential analysis, note that the count must be increasing at each time point. required: - data description: >- Run the analysis from sequential summary data. description: >- Data for the non-parametric sequential ratio test. description: Method-specific data for the group. required: - group - data description: >- Method-specific data for each group, note it has to be compatible with the method specified by analysis params. labels: type: array items: type: object additionalProperties: type: string description: Set of labels that will be returned in the result. description: The data for each segment. required: - id - segments description: Data for each hypothesis. securitySchemes: admin: type: http scheme: bearer ````