The Peeking Problem 2.0 (Part 1)
A new challenge in sequential testing with longitudinal data that can inflate false positive rates.
TL;DR
The standard peeking problem occurs when you analyze results before all participants have been observed, inflating false positive rates. Sequential testing solves this by accounting for repeated analyses.
But there's a second peeking problem. When each participant is measured at multiple points in time during the experiment, you can get substantially inflated false positive rates even when using sequential tests. The issue: sequential tests assume a unit is fully observed when it enters the analysis—but with longitudinal data, that's not true. A user who entered the experiment yesterday hasn't accumulated their full week of behavior yet. This post explains the problem and sets up the solution in Part 2.
Read the full post on Spotify Engineering: Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0