Configure Sequential Testing - Confidence Documentation

Sequential tests allow you to analyze experiment results during data collection without invalidating statistical conclusions. The Stats API uses the group sequential test method (gstZTest) for sequential analysis.

Fixed Horizon vs Sequential Tests

The Stats API supports two testing approaches:

Method	Description	When to Use
`zTest`	Fixed horizon z-test	When you analyze results once at the end of the experiment
`gstZTest`	Group sequential z-test	When you want to analyze results continuously during data collection

Configure a Group Sequential Test

Group sequential tests provide valid statistical conclusions even when you analyze results multiple times during data collection. You need to specify the expected maximum sample size upfront. In your analysis plan, specify gstZTest in the hypothesis segments:

{
  "hypotheses": [
    {
      "id": "my-metric",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "gstZTest": {
              "maxSampleSize": 10000
            }
          }
        }
      ]
    }
  ]
}

The maxSampleSize parameter helps the test allocate the false positive rate optimally across analyses. Your estimate doesn’t need to be exact, but a reasonable estimate improves power.

Configure a Fixed Horizon Test

For metrics you only analyze once at the end of the experiment, use the standard zTest:

{
  "hypotheses": [
    {
      "id": "my-metric",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "zTest": {}
          }
        }
      ]
    }
  ]
}

Provide Time-Series Data

Sequential tests require time-series data to track how results evolve. Structure your data with timeLabel values containing cumulative statistics up to each time point:

{
  "id": "my-metric",
  "segments": [
    {
      "dimensions": {},
      "groups": [
        {
          "group": "control",
          "data": {
            "gstZTest": {
              "summary": {
                "data": [
                  {
                    "timeLabel": "2024-01-01",
                    "value": {
                      "mean": 2.0,
                      "variance": 2.28,
                      "count": 100
                    }
                  },
                  {
                    "timeLabel": "2024-01-02",
                    "value": {
                      "mean": 1.99,
                      "variance": 2.29,
                      "count": 200
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  ]
}

Sequential Tests

Learn about sequential testing concepts

Run an Analysis

Complete analysis tutorial

Create Analysis Plan

Set up your analysis plan

Statistical Tests

Understand test types

​Fixed Horizon vs Sequential Tests

​Configure a Group Sequential Test

​Configure a Fixed Horizon Test

​Provide Time-Series Data

​Related Resources