The Experimentation RFP Series

Most platforms have a sample size calculator. Almost none connect it to the analysis the experiment will actually use. Here is what to ask vendors instead.

Every platform claims to support sequential testing. The claim is almost always incomplete. Here is what to ask vendors beyond the feature checklist.

Every platform lets you add multiple metrics. Most will display results for each one. What they rarely tell you is what to do next. Here is what to ask.

Every platform says it corrects for multiple comparisons. Most do, partially. Here is what to ask instead of "do you correct for multiple testing?"

Every platform offering variance reduction claims 20-50% runtime cuts. What is missing is how far the reduction reaches. Here is what to ask vendors.

Revenue per user. Streams per session. Most platforms support ratio metrics, and most get the variance wrong. Here is what to ask vendors instead.

Every platform estimates sample size before an experiment starts. Almost none revisit it after. Here is what to ask about in-flight power monitoring.

Every platform measures user behavior after exposure. The question most buyers never ask is: over what time period? Here is what to ask about observation windows.

Every platform lets you look at results. The question is whether the platform looks for you. Here is what to ask about monitoring and alerting.

Most platforms let you randomize by account or store. The question is what happens in the analysis after randomization. Here is what to ask.

When a user generates zero events, the platform makes a choice that changes what the experiment measures. Most vendors do not document which choice they make.

Every platform lets you slice by dimension. The question is whether the platform controls the false positive rate when you do.

Every platform lets you configure an experiment. Few require structured design review before launch. Here is what to ask vendors about review workflows.

Most experiments can run overlapping. For the ones that cannot, precise control matters. Here is what to ask beyond "do you support mutual exclusion?"

Most vendors support percentile metrics. Most implementations break down exactly when you need them. Here is what to ask instead of "do you support it?"

Most experimentation programs do not need geo-lift. If yours does, ask whether the platform confronts the assumptions that make or break the analysis.

Switchback experiments exist because standard A/B tests break down when users interact with each other. Only two vendors offer support. Here is what to ask.

The label Bayesian does not tell you what stopping rule is used. Here is what to ask instead.