Skip to content
  • Pricing
  • Contact us
  • Login
Start free
July 25, 2023· Updated May 12, 2026/Sebastian Ankargren, Mattias Frånberg, Mårten Schultzberg

The Peeking Problem 2.0 (Part 2): Sequential Testing

Sequential testing with longitudinal data can inflate false positive rates. Part 2 covers modeling approaches that handle repeated observations correctly.

Statistical chart on a dark background showing three user metric lines and confidence bounds illustrating the peeking problem

Start free with Confidence. No credit card needed.

Start free

#TL;DR

Part 1 introduced the peeking problem 2.0: when using longitudinal data, sequential tests can inflate false positive rates because units aren't fully observed when they enter the analysis.

The mechanisms are different from the original peeking problem, but the consequence is the same: inflated false positive rates. This post presents modeling approaches to handle multiple observations per experimental unit correctly. The key insight is that you need to account for the fact that users who recently entered the experiment contribute less information than users who have been in longer.

Read the full post on Spotify Engineering: Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing

Learn about experimentation from the free online Confidence bootcamp. This 72-lesson bootcamp has taught thousands of people at Spotify and other companies to experiment. Start learning right now!

Start free with Confidence. No credit card needed.

Start free

Read next

  • Fixed-Power Designs: It's Not IF You Peek, It's WHAT You Peek at

    A new experimental design that lets you estimate sample size during the experiment without compromising inference.

  • Choosing a Sequential Testing Framework — Comparisons and Discussions

    Sequential testing frameworks compared through simulation: group sequential tests, always-valid inference, and how your data infrastructure drives the choice.

  • Accurate Sample Size for Always-Valid Inference

    Most A/B testing tools either overestimate the sample size needed for always-valid inference or do not adjust for the sequential test in the sample size calculation at all. We derived a closed-form correction that requires no simulation.

PreviousExperiment like Spotify: With ConfidenceNextThe Peeking Problem 2.0 (Part 1)
Spotify

Product

  • Feature flags
  • Confidence Loop
  • Product Analytics
  • Session Recordings
  • Pricing
  • Comparisons

Solutions

  • Startups
  • Scale-ups
  • Enterprise
  • Engineers
  • Data scientists
  • Product managers
  • Data warehouse teams
  • Security

Resources

  • Docs
  • Blog
  • Bootcamp
  • Glossary
  • Why not Bayes?
  • RFP guides
  • Playlist (opens in a new tab)

Company

  • About
  • Contact us
  • Status (opens in a new tab)

Legal

  • Terms of Service
  • Data Protection Agreement
  • Privacy Policy
  • Security

© 2026 Spotify