Skip to content
Confidence
  • Documentation
  • Blog
  • Bootcamp
  • Status
  • Confidence Bootcamp
    • My learning
    • Intro to experimentation
      • Introduction
      • Lesson 1: Why you should experiment
      • Lesson 2: Experiment hypothesis
      • Lesson 3: Success and guardrail metrics
      • Lesson 4: Success metrics
      • Lesson 5: Set up your experiment
      • Lesson 6: Calculation frequency
      • Lesson 7: Target audience
      • Lesson 8: Sample size
      • Lesson 9: Quality assurance
      • Lesson 10: Run your experiment
      • Lesson 11: Evaluate your experiment and make a decision
      • Lesson 12: A/B tests and rollouts
      • Course wrap up
    • Intro to metrics
      • Introduction
      • Lesson 1: What is a metric?
      • Lesson 2: Metric roles
      • Lesson 3: Time considerations
      • Lesson 4: Capturing behavior
      • Lesson 5: Strategic metrics
      • Lesson 6: Interpretability
      • Lesson 7: Feasibility and sensitivity
      • Lesson 8: Variance reduction and metric selection
      • Lesson 9: Select metrics
      • Lesson 10: Segment-level analysis
      • Course wrap up
    • Scientific product development
      • Introduction
      • Lesson 1: Why you should experiment
      • Lesson 2: The scientific method
      • Lesson 3: Randomized controlled trials
      • Lesson 4: Experiment hypothesis
      • Lesson 5: Case study
        • Case study
        • Answers to case study
      • Lesson 6: Why do we need statistics?
      • Lesson 7: Success metrics
      • Lesson 8: Detectable effects and sample size
      • Lesson 9: Make a decision
      • Course wrap up
    • A primer on hypothesis testing
      • Introduction
      • Lesson 1: Introduction to hypothesis testing
      • Lesson 2: True vs estimated effects
      • Lesson 3: Sampling distribution of the difference-in-means estimator
      • Lesson 4: Z-tests and how to reject the null hypothesis
      • Lesson 5: False postive rate and alpha
      • Lesson 6: True positive rate, MDE, and power
      • Course wrap up
    • Intro to Feature Flags
      • Introduction
      • Lesson 1: What is a feature flag?
      • Lesson 2: Lifecycle of a feature flag
      • Lesson 3: Clients
      • Lesson 4: Evaluation context and targeting
    • Sample size calculation - I
      • Introduction
      • Lesson 1: What is the required sample size?
      • Lesson 2: Alpha and power
      • Lesson 3: Baseline mean and variance
      • Lesson 4: Sample size playground - I
    • Sample size calculation - II
      • Introduction
      • Lesson 1: Multi-metric decision making
      • Lesson 2: Number of success metrics
      • Lesson 3: Number of guardrail metrics
      • Lesson 4: Number of comparisons
      • Lesson 5: Sample size playground - II
    • Sample size calculation - III
      • Introduction
      • Lesson 1: Binary metrics
      • Lesson 2: Treatment group proportions
      • Lesson 3: Variance reduction
      • Lesson 4: Sequential testing and sample size
      • Lesson 5: Sample size playground - III
    • Advance your experimentation
      • Introduction
      • Lesson 1: Guardrail metrics with non-inferiority margins
      • Lesson 2: Choose evaluation frequency
      • Lesson 3: Metrics' roles in experiments
      • Lesson 4: Cumulative holdback evaluations
    • Experimentation culture
      • Introduction
      • Lesson 1: Onboarding into experimentation
      • Lesson 2: Empowering experimentation champions
      • Lesson 3: Sustaining the experimentation culture
    • Videos

Lesson 5: Sample size calculation playground - level III

You have made it to the final boss. In this lesson, you get to play around with all the parameters that affect the sample size calculation in our calculator to get a feel for how the different parameters affect the sample size.

Summary

This is the third and final level, so now you have all the parameters from all three levels of this course free for you to play around with!

The idea with this lesson is simple: you change the parameters in the sample size calculator and see how the sample size changes. In this lesson, there are more questions than in other lessons, and they can be answered by experimenting with the sample size calculator.

Recommendation

Turn on the 'Show detailed formulas' option in the sample size calculator to see the formulas used to calculate the sample size. This can help you understand how the different parameters affect the sample size.

Required sample size calculator (level 3)

Metric parameters

0

Evaluate once, upon conclusion of the experiment.

Statistical parameters

Experiment design

1
1
0
0

Sample Allocation

50
50

Required Sample Size: 0

Reader exercise

What happens to the required sample size when you increase the baseline mean for a binary metric from 0.2 to 0.8?

Reader exercise

How does reducing the Relative MDE/NIM (%) from 5% to 1% affect the required sample size?

Reader exercise

How does the required sample size differ between a binary metric with a baseline mean of 0.5 and a continuous metric with a baseline mean of 0.5?

Reader exercise

What is the effect of increasing the Variance Reduction Factor from 0% to 50% on the required sample size?

Reader exercise

How does reducing the alpha (false positive rate) from 0.1 to 0.01 impact the required sample size?

Reader exercise

What happens to the required sample size when increasing the power (true positive rate) from 80% to 95%?

Reader exercise

How does increasing the number of comparisons from 1 to 5 impact the required sample size?

Reader exercise

What happens to the required sample size and group allocation when you change the control group proportion from 50% to 80%?

Reader exercise

How does increasing the number of success metrics from 1 to 3 affect the required sample size?

Reader exercise

What is the effect of increasing the number of guardrail metrics with NIMs from 1 to 5 on the required sample size?

Reader exercise

What is the effect of increasing the number of guardrail metrics without NIMs from 0 to 5 on the required sample size for the shipping decision?

Was this page helpful?

PreviousLesson 4: Sequential testing and sample size
NextIntroduction

© Copyright 2026. All rights reserved.

Follow us on TwitterFollow us on GitHub