Skip to main content
Sequential tests allow you to analyze experiment results during data collection without invalidating statistical conclusions. The Stats API uses the group sequential test method (gstZTest) for sequential analysis.

​
Fixed Horizon vs Sequential Tests

The Stats API supports two testing approaches:
MethodDescriptionWhen to Use
zTestFixed horizon z-testWhen you analyze results once at the end of the experiment
gstZTestGroup sequential z-testWhen you want to analyze results continuously during data collection

​
Configure a Group Sequential Test

Group sequential tests provide valid statistical conclusions even when you analyze results multiple times during data collection. You need to specify the expected maximum sample size upfront. In your analysis plan, specify gstZTest in the hypothesis segments: 
{
  "hypotheses": [
    {
      "id": "my-metric",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "gstZTest": {
              "maxSampleSize": 10000
            }
          }
        }
      ]
    }
  ]
}
The maxSampleSize parameter helps the test allocate the false positive rate optimally across analyses. Your estimate doesn’t need to be exact, but a reasonable estimate improves power.

​
Configure a Fixed Horizon Test

For metrics you only analyze once at the end of the experiment, use the standard zTest: 
{
  "hypotheses": [
    {
      "id": "my-metric",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "zTest": {}
          }
        }
      ]
    }
  ]
}

​
Provide Time-Series Data

Sequential tests require time-series data to track how results evolve. Structure your data with timeLabel values containing cumulative statistics up to each time point: 
{
  "id": "my-metric",
  "segments": [
    {
      "dimensions": {},
      "groups": [
        {
          "group": "control",
          "data": {
            "gstZTest": {
              "summary": {
                "data": [
                  {
                    "timeLabel": "2024-01-01",
                    "value": {
                      "mean": 2.0,
                      "variance": 2.28,
                      "count": 100
                    }
                  },
                  {
                    "timeLabel": "2024-01-02",
                    "value": {
                      "mean": 1.99,
                      "variance": 2.29,
                      "count": 200
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  ]
}

Sequential Tests

Learn about sequential testing concepts

Run an Analysis

Complete analysis tutorial

Create Analysis Plan

Set up your analysis plan

Statistical Tests

Understand test types