Choosing a Sequential Testing Framework — Comparisons and Discussions
Sequential testing frameworks compared through simulation: group sequential tests, always-valid inference, and how your data infrastructure drives the choice.
Want to experiment like Spotify? Sign up for a 30 day free trial.
Start your free trialTL;DR
Sequential tests let you analyze experiments while data is still being collected without inflating false positive rates. But which sequential test should you use? The literature has developed quickly, and most leading A/B testing companies have their own favorite.
This post compares different sequential testing frameworks using simulation results. Two main parameters should affect your choice: whether your data infrastructure provides data in batch or streaming, and whether you can estimate the maximum sample size upfront. Spotify uses group sequential tests (GSTs) because they were originally designed for medical studies where data arrived in batches—similar to how our data infrastructure works.
Read the full post on Spotify Engineering: Choosing a Sequential Testing Framework — Comparisons and Discussions
Learn about experimentation from the free online Confidence bootcamp. This 72-lesson bootcamp has taught thousands of people at Spotify and other companies to experiment. Start learning right now!