# Confidence

> Confidence is Spotify's warehouse-native experimentation and feature management platform. It provides feature flags, A/B testing, rollouts, metrics, and statistical analysis to help organizations run experiments and make data-informed product decisions — all on top of your own data warehouse.

Confidence is built on years of research and experience running large experimentation programs at Spotify. A/B tests help evaluate new ideas, with data informing decisions about what and how to build. Rollouts enable safe releases of features and backend changes. In both cases, the focus is on learning from users and controlling risks.

Confidence is warehouse-native: it runs experiments directly in your existing data warehouse (BigQuery, Snowflake, Redshift, or Databricks), giving you full data transparency and keeping data in an environment you control.

The platform uses OpenFeature-compliant SDKs from the Cloud Native Computing Foundation (CNCF), reducing vendor lock-in.

You can [sign up for a free trial](https://app.confidence.spotify.com/signup) to get started — no credit card needed.

Trusted by teams at Spotify, Truecaller, Soundtrack, Dott, and Mentimeter.

## Product features

### Feature flags

Create and manage feature flags from the CLI, Slack, your IDE (VS Code, Cursor), or the Confidence UI. Ship to who you want, when you want. Confidence supports unlimited feature flags on every plan, including the free tier. Flags use the OpenFeature standard from CNCF, so there is no vendor lock-in. SDKs cover server and client platforms, with support for local evaluation, edge resolvers, and dynamic configs that carry structured data instead of simple booleans.

### A/B testing and experimentation

Run A/B tests with automatic metric analysis. Define a hypothesis, set success and guardrail metrics, and Confidence handles the statistical analysis — sequential tests, CUPED variance reduction, multiple testing corrections, and power analyses. Results are presented in automated dashboards with clear statistical conclusions. Confidence supports A/B tests, A/B/n tests, and analysis of experiments conducted in external systems.

### Safe rollouts

Release features in controlled stages with automatic monitoring. Define guardrail metrics that represent what "safe" looks like. Confidence monitors those metrics, catches regressions the moment they happen, and rolls back automatically before user experience degrades. Supports automatic percentage ramp-up on your schedule, with the ability to pause or roll back instantly from anywhere.

### Flag lifecycle management

Once a flag reaches 100%, Confidence detects it, opens a pull request to remove the dead code, and gets out of your way. This keeps codebases clean without manual tracking.

### Warehouse-native analytics

Confidence runs experiments directly in your existing data warehouse — Snowflake, BigQuery, Databricks, or Redshift. Define metrics in SQL, reuse existing data models, and keep your warehouse as the source of truth. Every query Confidence runs is transparent and auditable. Warehouse-native is included on every plan, starting with Free.

### Integrations

Confidence works where you work: Slack, CLI, VS Code, Cursor, Claude (via MCP), and the web UI. Manage flags, review experiments, and trigger rollbacks without switching tools.

### Dashboards and analysis

Automated analytics dashboards for each experiment, designed to be easy for anyone to use while being sophisticated enough to power analysis across all experiments at Spotify. Built-in statistical methods include sequential tests, CUPED, multiple testing corrections, and one-click power analyses and sample size calculations.

### Adoption and culture tools

Self-serve onboarding materials and courses (the same ones that onboarded thousands of experimenters at Spotify). Customizable experience that hides advanced options for less seasoned teams. Progress tracking for experimentation adoption and velocity directly in the tool, with an insights page showing decision quality.

## Use cases

### Validate product changes with A/B tests

Test new features, UI changes, and product ideas by running controlled experiments. Compare treatment and control groups with rigorous statistical analysis to know what works before committing to it. At Spotify, this approach validated features like Discover Weekly, where A/B testing proved the concept despite initial skepticism.

### Release features safely with rollouts

Ship backend changes, infrastructure updates, and new features with controlled rollouts. Define guardrail metrics, set ramp-up schedules, and let Confidence monitor for regressions. Auto-rollback catches problems before they reach your full user base.

### Measure cumulative product impact

Use holdback groups to measure the aggregate impact of many shipped features over time, maintaining a long-running holdout group that does not receive new features.

### Scale experimentation across your organization

Move experimentation from a specialized practice into a company-wide habit. Confidence provides self-serve onboarding, guided workflows for new experimenters, and progress tracking to measure adoption across teams.

### Analyze experiments from external systems

Bring experiment data from external systems into Confidence for analysis. Use the warehouse-native architecture to run statistical analysis on experiments conducted elsewhere, without duplicating event tracking.

## Pricing

For structured pricing data (plans, limits, usage rules, terms), see: [pricing.md](https://confidence.spotify.com/pricing.md)

Confidence uses credit-based pricing. One credit = one unique user exposed to an experiment in a given month. Feature flags are free at any scale. Free plan at $0/month (75,000 credits), Growth from $449/month (up to 2,500,000 credits), Enterprise custom. 30-day free trial on Growth, no credit card required.

## Docs

For a comprehensive LLM-optimized version of the documentation, see: [Confidence Docs llms.txt](https://confidence.spotify.com/docs/llms.txt)

- [Introduction to Confidence](https://confidence.spotify.com/docs/introduction): Overview of the platform and its capabilities
- [Warehouse-Native Confidence](https://confidence.spotify.com/docs/data-warehouse-native): Run experiments on top of your own data warehouse
- [Confidence as a Platform](https://confidence.spotify.com/docs/confidence-as-a-platform): Platform architecture overview

## Getting started

- [Onboarding Guide](https://confidence.spotify.com/docs/onboarding-guide): Step-by-step guide to get started
- [BigQuery Setup](https://confidence.spotify.com/docs/warehouse-setup/bigquery): Connect BigQuery as your data warehouse
- [Snowflake Setup](https://confidence.spotify.com/docs/warehouse-setup/snowflake): Connect Snowflake as your data warehouse
- [Redshift Setup](https://confidence.spotify.com/docs/warehouse-setup/redshift): Connect Redshift as your data warehouse
- [Databricks Setup](https://confidence.spotify.com/docs/warehouse-setup/databricks): Connect Databricks as your data warehouse

## Quickstarts

- [Configure a Flag](https://confidence.spotify.com/docs/quickstarts/configure-flag): Create and configure your first feature flag
- [Configure a Metric](https://confidence.spotify.com/docs/quickstarts/configure-metric): Define metrics to measure experiment impact
- [Launch a Rollout](https://confidence.spotify.com/docs/quickstarts/launch-rollout): Safely release changes with a rollout
- [Launch an A/B Test](https://confidence.spotify.com/docs/quickstarts/launch-abtest): Set up and run an A/B test
- [Analyze a Past Experiment](https://confidence.spotify.com/docs/quickstarts/analyze-past-experiment): Analyze experiments conducted elsewhere

## Feature flags

- [Flags Introduction](https://confidence.spotify.com/docs/flags/introduction): Overview of feature flags in Confidence
- [Clients](https://confidence.spotify.com/docs/flags/clients): SDK clients for resolving flags
- [Context Schemas](https://confidence.spotify.com/docs/flags/context-schema): Define context used for flag evaluation
- [Audiences](https://confidence.spotify.com/docs/flags/audience): Target specific user groups
- [Segments](https://confidence.spotify.com/docs/flags/segments): Reusable targeting criteria
- [Rules](https://confidence.spotify.com/docs/flags/define-rules): Configure flag evaluation rules
- [Resolve Tester](https://confidence.spotify.com/docs/how-to-guides/test-flag-resolution): Test flag resolution behavior
- [Local Resolver](https://confidence.spotify.com/docs/how-to-guides/setup-local-resolver): Resolve flags locally for development

## Metrics

- [Metrics Introduction](https://confidence.spotify.com/docs/metrics/introduction): Overview of the metrics system
- [Entities](https://confidence.spotify.com/docs/metrics/entities): Define the units of analysis
- [Assignment Tables](https://confidence.spotify.com/docs/metrics/assignment-tables): Map entities to experiment treatments
- [Exposure](https://confidence.spotify.com/docs/metrics/exposure): Track who is exposed to an experiment
- [Fact Tables](https://confidence.spotify.com/docs/metrics/fact-tables): Define data sources for metrics
- [Dimension Tables](https://confidence.spotify.com/docs/metrics/dimension-tables): Add dimensions for metric breakdowns
- [Metrics](https://confidence.spotify.com/docs/metrics/metrics): Define and manage metrics

## Experiments

- [Experiments Introduction](https://confidence.spotify.com/docs/experiments/introduction): Overview of experimentation in Confidence
- [A/B Tests](https://confidence.spotify.com/docs/experiments/workflows/abtests): Compare variants to measure impact
- [Rollouts](https://confidence.spotify.com/docs/experiments/workflows/rollouts): Gradually release changes with monitoring
- [Analysis Workflows](https://confidence.spotify.com/docs/experiments/workflows/analysis): Analyze experiments from external systems
- [Treatments](https://confidence.spotify.com/docs/experiments/treatments): Configure experiment variants
- [Audience](https://confidence.spotify.com/docs/experiments/audience): Define who participates in an experiment
- [Exclusive Experiments](https://confidence.spotify.com/docs/experiments/exclusive-experiments): Prevent experiment interactions
- [Statistical Settings](https://confidence.spotify.com/docs/experiments/statistical-settings): Configure significance levels and test parameters
- [Sample Size Calculator](https://confidence.spotify.com/docs/experiments/sample-size-calculator): Estimate required sample size
- [Monitoring](https://confidence.spotify.com/docs/experiments/monitoring): Track experiment health in real time
- [Analyze Results](https://confidence.spotify.com/docs/experiments/analyze-results): Interpret experiment outcomes

## Statistics

- [Hypothesis Testing](https://confidence.spotify.com/docs/experiments/design/hypothesis): Formulate experiment hypotheses
- [Alpha and Power](https://confidence.spotify.com/docs/experiments/design/power): Configure type I and type II error rates
- [Effect Sizes](https://confidence.spotify.com/docs/experiments/design/effect-sizes): Define minimum detectable effects
- [Metrics and Tests](https://confidence.spotify.com/docs/experiments/stats/stat-tests): Statistical tests for different metric types
- [Multiple Comparisons](https://confidence.spotify.com/docs/experiments/stats/adjustment-multiple-comparisons): Adjust for testing multiple metrics
- [Sequential Tests](https://confidence.spotify.com/docs/experiments/stats/sequential-tests): Monitor experiments with valid early stopping
- [Variance Reduction (CUPED)](https://confidence.spotify.com/docs/experiments/stats/variance-reduction): Reduce variance using pre-experiment data

## APIs

- [API Quickstart](https://confidence.spotify.com/docs/api/quickstart): Get started with the Confidence API
- [API Basics](https://confidence.spotify.com/docs/api/api-basics): Authentication, pagination, and common patterns
- [Flags API](https://confidence.spotify.com/docs/api/flags/concepts): Manage and resolve feature flags programmatically
- [Metrics API](https://confidence.spotify.com/docs/api/metrics/concepts): Define entities, tables, and metrics via API
- [Stats API](https://confidence.spotify.com/docs/api/stats/concepts): Run statistical analyses and power analyses
- [Experiments API](https://confidence.spotify.com/docs/api/experiments): Manage A/B tests and rollouts via API
- [IAM API](https://confidence.spotify.com/docs/api/iam): Manage users, API clients, and access control

## SDKs

- [SDK Introduction](https://confidence.spotify.com/docs/sdks/introduction): Overview of available SDKs and integrations
- [Context](https://confidence.spotify.com/docs/sdks/context): Set evaluation context for flag resolution
- [Apply Event](https://confidence.spotify.com/docs/sdks/apply-event): Track when a flag value is applied

## Identity and access management

- [IAM Introduction](https://confidence.spotify.com/docs/iam/introduction): Overview of identity and access management
- [Users](https://confidence.spotify.com/docs/iam/users): Manage user accounts
- [Groups](https://confidence.spotify.com/docs/iam/groups): Organize users into groups
- [Roles](https://confidence.spotify.com/docs/iam/roles): Define roles with specific permissions
- [Policies](https://confidence.spotify.com/docs/iam/policies): Configure access policies

## MCP servers

- [MCP Servers](https://confidence.spotify.com/docs/sdks/mcp-servers): Use Confidence with AI assistants via Model Context Protocol

## Migration guides

- [Migrate to Confidence](https://confidence.spotify.com/docs/migrations/overview): Migrate feature flags and experiments from PostHog, Eppo, Statsig, or Optimizely to Confidence using AI-powered migration kits
- [Migrate from Statsig](https://confidence.spotify.com/docs/migrations/migrate-from-statsig): Migrate feature gates, dynamic configs, and experiments from Statsig to Confidence with the AI-powered migration kit
- [Migrate from PostHog](https://confidence.spotify.com/docs/migrations/migrate-from-posthog): Migrate feature flags and multivariate flags from PostHog to Confidence with the AI-powered migration kit
- [Migrate from Eppo](https://confidence.spotify.com/docs/migrations/migrate-from-eppo): Migrate feature flags, allocations, and audiences from Eppo to Confidence with the AI-powered migration kit
- [Migrate from Optimizely](https://confidence.spotify.com/docs/migrations/migrate-from-optimizely): Migrate flags, rules, and audiences from Optimizely Feature Experimentation to Confidence with the AI-powered migration kit

## Comparisons

Side-by-side comparisons of Confidence with other experimentation and feature management platforms, plus "what is" overviews and alternatives lists. Each page is also available as plain markdown at the same URL with `.md` appended.

### Confidence vs competitors

- [Confidence vs Statsig](https://confidence.spotify.com/comparisons/confidence-vs-statsig): Head-to-head comparison of warehouse-native experimentation platforms
- [Confidence vs Eppo](https://confidence.spotify.com/comparisons/confidence-vs-eppo): Both run analysis in your warehouse — the difference is operating history and metric workflow
- [Confidence vs LaunchDarkly](https://confidence.spotify.com/comparisons/confidence-vs-launchdarkly): Feature-flag-first vs experimentation-first platform architectures
- [Confidence vs Optimizely](https://confidence.spotify.com/comparisons/confidence-vs-optimizely): Experimentation-first vendor vs content-and-commerce suite with experimentation
- [Confidence vs GrowthBook](https://confidence.spotify.com/comparisons/confidence-vs-growthbook): Managed platform with opinionated defaults vs open-source self-hosted flexibility
- [Confidence vs PostHog](https://confidence.spotify.com/comparisons/confidence-vs-posthog): Experimentation depth vs bundled product analytics suite
- [Confidence vs Split (Harness FME)](https://confidence.spotify.com/comparisons/confidence-vs-split): Comparison with Split, now Harness Feature Management & Experimentation

### What is each platform?

- [What is Statsig?](https://confidence.spotify.com/comparisons/what-is-statsig): Overview of Statsig's feature flagging, experimentation, and product analytics platform
- [What is Eppo?](https://confidence.spotify.com/comparisons/what-is-eppo): Overview of Eppo's warehouse-native experimentation platform
- [What is LaunchDarkly?](https://confidence.spotify.com/comparisons/what-is-launchdarkly): Overview of LaunchDarkly's enterprise feature flag platform
- [What is Optimizely?](https://confidence.spotify.com/comparisons/what-is-optimizely): Overview of Optimizely's Digital Experience Platform
- [What is GrowthBook?](https://confidence.spotify.com/comparisons/what-is-growthbook): Overview of GrowthBook's open-source experimentation platform
- [What is Split?](https://confidence.spotify.com/comparisons/what-is-split): Overview of Split (now Harness FME) feature flag and experimentation platform

### Alternatives lists

- [Alternatives to Statsig](https://confidence.spotify.com/comparisons/alternatives-to-statsig): Top alternatives to Statsig for experimentation
- [Alternatives to Eppo](https://confidence.spotify.com/comparisons/alternatives-to-eppo): Top alternatives to Eppo for experimentation
- [Alternatives to LaunchDarkly](https://confidence.spotify.com/comparisons/alternatives-to-launchdarkly): Top alternatives to LaunchDarkly for feature flags and experimentation
- [Alternatives to Optimizely](https://confidence.spotify.com/comparisons/alternatives-to-optimizely): Top alternatives to Optimizely for experimentation
- [Alternatives to GrowthBook](https://confidence.spotify.com/comparisons/alternatives-to-growthbook): Top alternatives to GrowthBook for experimentation
- [Alternatives to PostHog](https://confidence.spotify.com/comparisons/alternatives-to-posthog): Top alternatives to PostHog for experimentation
- [Alternatives to Split](https://confidence.spotify.com/comparisons/alternatives-to-split): Top alternatives to Split (Harness FME) for experimentation

## Bootcamp

The Confidence Bootcamp is a free learning program covering experimentation, metrics, hypothesis testing, sample size calculation, and experimentation culture. Each course contains sequential lessons.

For the complete full-text content of all bootcamp lessons, see: [Confidence Bootcamp llms-full.txt](https://confidence.spotify.com/bootcamp/llms-full.txt)

### Frequently Asked Questions

- **Who is the Confidence Bootcamp for?** The bootcamp is designed for anyone who wants to improve their experimentation skills — data scientists, analysts, engineers, product managers, and leaders.
- **Is the bootcamp free?** Yes, completely free. All 11 courses, 90+ lessons, and resources are available at no cost.
- **What will I learn?** A/B testing fundamentals, hypothesis formulation, interpreting experiment results, metrics design, sample size calculation, feature flags, and building an experimentation culture.
- **How long does it take?** Approximately 20 hours across all 11 courses. Individual courses range from 30 minutes to 3 hours.
- **Do I need prior experience?** No prior experience is required. The bootcamp starts with foundational courses and progressively covers advanced topics.
- **Who created it?** The Confidence team at Spotify, the same team that builds the experimentation and feature flagging platform used across Spotify.

### Intro to experimentation

- [Introduction](https://confidence.spotify.com/bootcamp/intro-course/introduction): Course overview
- [Why you should experiment](https://confidence.spotify.com/bootcamp/intro-course/why-you-should-experiment): The case for experimentation — biases, accidental breakage, and causal conclusions
- [Experiment hypothesis](https://confidence.spotify.com/bootcamp/intro-course/experiment-hypothesis): How to formulate a clear experiment hypothesis
- [Success and guardrail metrics](https://confidence.spotify.com/bootcamp/intro-course/success-and-guardrail-metrics): How to measure impact with success and guardrail metrics
- [Success metrics](https://confidence.spotify.com/bootcamp/intro-course/success-metrics): Choosing the right success metrics
- [Set up your experiment](https://confidence.spotify.com/bootcamp/intro-course/set-up-your-experiment): Experiment setup in Confidence
- [Calculation frequency](https://confidence.spotify.com/bootcamp/intro-course/calculation-frequency): How often experiment results are calculated
- [Target audience](https://confidence.spotify.com/bootcamp/intro-course/target-audience): Defining who participates in an experiment
- [Sample size](https://confidence.spotify.com/bootcamp/intro-course/sample-size): Determining required sample size
- [Quality assurance](https://confidence.spotify.com/bootcamp/intro-course/quality-assurance): QA checks before and during experiments
- [Run your experiment](https://confidence.spotify.com/bootcamp/intro-course/run-your-experiment): Monitoring and managing a running experiment
- [Evaluate and decide](https://confidence.spotify.com/bootcamp/intro-course/evaluate-your-experiment-and-make-a-decision): Interpreting results and making decisions
- [A/B tests and rollouts](https://confidence.spotify.com/bootcamp/intro-course/ab-tests-and-rollouts): Differences between A/B tests and rollouts

### Interpreting experiment results

- [Introduction](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/introduction): Course overview
- [Anatomy of the results page](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/the-anatomy-of-the-results-page): Understanding the results page layout
- [The Spotlight](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/the-spotlight): How to use the Spotlight summary
- [Means and relative effects](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/reading-means-and-relative-effects): Reading means and relative effects
- [Confidence intervals and precision](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/confidence-intervals-and-precision): Understanding confidence intervals
- [Significance for success metrics](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/significance-and-status-labels): Significance labels and what they mean
- [Guardrail metrics and NIMs](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/guardrail-metrics-and-nims): Non-inferiority margins for guardrail metrics
- [Health checks and SRM](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/health-checks-and-srm): Sample ratio mismatch and experiment health
- [Variance reduction](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/variance-reduction): CUPED and variance reduction techniques
- [Sequential vs non-sequential](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/sequential-vs-non-sequential): When to use sequential testing
- [Exploratory analysis](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/exploratory-analysis): Dimension slicing and exploratory analysis
- [The winner's curse](https://confidence.spotify.com/bootcamp/interpreting-experiment-results/type-m-and-type-s-errors): Type M and Type S errors

### Intro to metrics

- [Introduction](https://confidence.spotify.com/bootcamp/intro-to-metrics/introduction): Course overview
- [What is a metric?](https://confidence.spotify.com/bootcamp/intro-to-metrics/what-is-a-metric): Defining metrics for experimentation
- [Metric roles](https://confidence.spotify.com/bootcamp/intro-to-metrics/metric-roles): Success, guardrail, and monitoring metrics
- [Time considerations](https://confidence.spotify.com/bootcamp/intro-to-metrics/time-considerations): How time affects metric measurement
- [Capturing behavior](https://confidence.spotify.com/bootcamp/intro-to-metrics/capturing-behavior): Translating user behavior into measurable metrics
- [Strategic metrics](https://confidence.spotify.com/bootcamp/intro-to-metrics/strategic-metrics): Aligning metrics with business strategy
- [Interpretability](https://confidence.spotify.com/bootcamp/intro-to-metrics/interpretability): Making metrics easy to understand and act on
- [Feasibility and sensitivity](https://confidence.spotify.com/bootcamp/intro-to-metrics/feasibility-and-sensitivity): Practical constraints on metric selection
- [Variance reduction](https://confidence.spotify.com/bootcamp/intro-to-metrics/variance-reduction): Reducing noise in metric measurements
- [Select metrics](https://confidence.spotify.com/bootcamp/intro-to-metrics/select-metrics): Choosing the right set of metrics
- [Segment-level analysis](https://confidence.spotify.com/bootcamp/intro-to-metrics/segment-analysis): Breaking down results by user segments

### Scientific product development

- [Introduction](https://confidence.spotify.com/bootcamp/scientific-product-development/introduction): Course overview
- [Why you should experiment](https://confidence.spotify.com/bootcamp/scientific-product-development/why-you-should-experiment): The case for scientific product development
- [The scientific method](https://confidence.spotify.com/bootcamp/scientific-product-development/the-scientific-method): Origins of the scientific method applied to product
- [Randomized controlled trials](https://confidence.spotify.com/bootcamp/scientific-product-development/rct): Why randomized controlled experiments matter
- [Experiment hypothesis](https://confidence.spotify.com/bootcamp/scientific-product-development/experiment-hypothesis): Formulating experiment hypotheses
- [Case study: Shuffle button](https://confidence.spotify.com/bootcamp/scientific-product-development/shuffle-example-1): Case study — shuffle button in a shelf on Spotify Home
- [Why do we need statistics?](https://confidence.spotify.com/bootcamp/scientific-product-development/why-do-we-need-statistics): The role of statistics in experimentation
- [Success metrics](https://confidence.spotify.com/bootcamp/scientific-product-development/success-metrics): Choosing success metrics for experiments
- [Detectable effects and sample size](https://confidence.spotify.com/bootcamp/scientific-product-development/detectable-effects-and-sample-size): Setting the MDE and calculating sample size
- [Make a decision](https://confidence.spotify.com/bootcamp/scientific-product-development/decision-making): Making ship/no-ship decisions from results

### A primer on hypothesis testing

- [Introduction](https://confidence.spotify.com/bootcamp/hypothesis-testing/introduction): Course overview
- [Introduction to hypothesis testing](https://confidence.spotify.com/bootcamp/hypothesis-testing/what-is-hypothesis-testing): What hypothesis tests are and why they matter
- [True vs estimated effects](https://confidence.spotify.com/bootcamp/hypothesis-testing/true-vs-estimated-effects): Understanding estimation error in experiments
- [Sampling distribution](https://confidence.spotify.com/bootcamp/hypothesis-testing/sampling-distribution-of-the-difference-in-means): Sampling distribution of the difference-in-means estimator
- [Z-tests and rejecting the null](https://confidence.spotify.com/bootcamp/hypothesis-testing/z-tests-and-rejecting-the-null): Z-tests and how to reject the null hypothesis
- [False positive rate and alpha](https://confidence.spotify.com/bootcamp/hypothesis-testing/fpr-and-alpha): Type I error rate and significance level
- [True positive rate and power](https://confidence.spotify.com/bootcamp/hypothesis-testing/tpr-and-power): True positive rate, MDE, and statistical power

### Sample size calculation - Level I

- [Introduction](https://confidence.spotify.com/bootcamp/sample-size-calculation-i/introduction): Course overview
- [Required sample size](https://confidence.spotify.com/bootcamp/sample-size-calculation-i/what-is-the-required-sample-size): What determines the required sample size
- [Alpha and power](https://confidence.spotify.com/bootcamp/sample-size-calculation-i/alpha-and-power): Refresher on alpha and power for sample size
- [Baseline mean and variance](https://confidence.spotify.com/bootcamp/sample-size-calculation-i/mean-variance-and-mde): How baseline mean, variance, and MDE affect sample size
- [Sample size playground I](https://confidence.spotify.com/bootcamp/sample-size-calculation-i/ssc-playground-i): Interactive sample size calculator

### Sample size calculation - Level II

- [Introduction](https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/introduction): Course overview
- [Multi-metric decisions](https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/multi-metric-decisions): Impact of multiple metrics on sample size
- [Number of success metrics](https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/number-of-success-metrics): How the number of success metrics affects sample size
- [Number of guardrail metrics](https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/number-of-guardrail-metrics): How guardrail metrics affect sample size
- [Number of comparisons](https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/number-of-comparisons): How multiple comparisons affect sample size
- [Sample size playground II](https://confidence.spotify.com/bootcamp/sample-size-calculation-ii/ssc-playground-ii): Interactive sample size calculator — level II

### Sample size calculation - Level III

- [Introduction](https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/introduction): Course overview
- [Binary metrics](https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/binary-metrics): Sample size for binary metrics
- [Treatment proportions](https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/treatment-proportion): Impact of unequal treatment group sizes
- [Variance reduction](https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/variance-reduction): How variance reduction reduces required sample size
- [Sample size playground III](https://confidence.spotify.com/bootcamp/sample-size-calculation-iii/ssc-playground-iii): Interactive sample size calculator — level III

### Advance your experimentation

- [Introduction](https://confidence.spotify.com/bootcamp/advancing-experimentation/introduction): Course overview
- [Guardrail metrics with NIMs](https://confidence.spotify.com/bootcamp/advancing-experimentation/guardrail-metrics-and-nims): Guardrail metrics with non-inferiority margins
- [Choose evaluation strategy](https://confidence.spotify.com/bootcamp/advancing-experimentation/choose-evaluation-strategy): Sequential vs non-sequential testing strategy
- [Metric roles in experiments](https://confidence.spotify.com/bootcamp/advancing-experimentation/metric-roles): Different types of metrics in experiments
- [Cumulative holdback evaluations](https://confidence.spotify.com/bootcamp/advancing-experimentation/holdbacks): Measuring cumulative impact with holdbacks

### Experimentation culture

- [Introduction](https://confidence.spotify.com/bootcamp/experimentation-culture/introduction): Building an experimentation-driven culture
- [Onboarding into experimentation](https://confidence.spotify.com/bootcamp/experimentation-culture/onboarding): Onboarding new colleagues to experimentation
- [Experimentation champions](https://confidence.spotify.com/bootcamp/experimentation-culture/champions): Empowering experimentation champions
- [Sustaining the culture](https://confidence.spotify.com/bootcamp/experimentation-culture/sustaining-the-culture): Maintaining experimentation culture long-term

### Working with feature flags

- [Introduction](https://confidence.spotify.com/bootcamp/intro-to-feature-flags/introduction): Course overview
- [What is a feature flag?](https://confidence.spotify.com/bootcamp/intro-to-feature-flags/what-is-a-feature-flag): Feature flag fundamentals
- [Lifecycle](https://confidence.spotify.com/bootcamp/intro-to-feature-flags/life-cycle): Lifecycle of a feature flag
- [Clients](https://confidence.spotify.com/bootcamp/intro-to-feature-flags/clients): Working with feature flag clients
- [Evaluation context and targeting](https://confidence.spotify.com/bootcamp/intro-to-feature-flags/evaluation-context-and-targeting): Context and targeting rules for feature flags

## RFP Guides

Detailed guides for evaluating experimentation platforms. Each guide covers what to ask vendors, what good answers look like, and what pitfalls to watch for when writing a Request for Proposal (RFP) for an experimentation platform.

- [RFP Guides Index](https://confidence.spotify.com/rfp): Browse all experimentation platform RFP guides
- [Sample Size Calculators](https://confidence.spotify.com/rfp/rfp-sample-size): What to ask about sample size estimation and whether the calculator connects to the actual analysis method
- [Sequential Testing](https://confidence.spotify.com/rfp/rfp-sequential-testing): What to ask about sequential testing beyond "do you have it" — stopping rules, alpha spending, and valid confidence intervals
- [Multi-Metric Decision Making](https://confidence.spotify.com/rfp/rfp-multi-metric): What to ask about handling multiple metrics — guardrails, trade-offs, and decision frameworks
- [Multiple Testing Corrections](https://confidence.spotify.com/rfp/rfp-multiple-testing): What to ask about correction methods — which comparisons are corrected and which are not
- [Variance Reduction](https://confidence.spotify.com/rfp/rfp-variance-reduction): What to ask about CUPED and other variance reduction methods — coverage, automation, and actual runtime savings
- [Ratio Metrics](https://confidence.spotify.com/rfp/rfp-ratio-metrics): What to ask about ratio metrics like revenue per user — variance estimation and the delta method
- [Fixed-Power Designs](https://confidence.spotify.com/rfp/rfp-fixed-power): What to ask about during-experiment power monitoring and what happens when assumptions change mid-flight
- [Observation Windows and Time-in Metrics](https://confidence.spotify.com/rfp/rfp-time-in-metrics): What to ask about how platforms handle observation periods and time-based metric windows
- [Monitoring and Alerting](https://confidence.spotify.com/rfp/rfp-monitoring-alerting): What to ask about automated experiment health checks, guardrail alerts, and sample ratio mismatch detection
- [Clustered Randomization](https://confidence.spotify.com/rfp/rfp-clustered-randomization): What to ask about cluster-level randomization and whether the analysis accounts for the clustering
- [Metric Zero-Handling](https://confidence.spotify.com/rfp/rfp-zero-handling): What to ask about how the platform treats users with zero events and the impact on metric definitions
- [Exploratory Analysis and Dimensions](https://confidence.spotify.com/rfp/rfp-exploratory-analysis): What to ask about dimension slicing and false positive rate control in exploratory analysis
- [Percentile Metrics](https://confidence.spotify.com/rfp/rfp-percentiles): What to ask about percentile metric support — bootstrap methods, confidence intervals, and where implementations break down
- [Experiment Design Review](https://confidence.spotify.com/rfp/rfp-experiment-review): What to ask about structured review of experiment design before launch
- [Experiment Coordination](https://confidence.spotify.com/rfp/rfp-experiment-coordination): What to ask about mutual exclusion, experiment interactions, and coordination across concurrent experiments
- [Bayesian Inference](https://confidence.spotify.com/rfp/rfp-bayesian): What to ask about Bayesian methods — priors, stopping rules, and what "Bayesian" actually means for the analysis
- [Geo-Lift and Synthetic Control](https://confidence.spotify.com/rfp/rfp-geo-lift): What to ask about geo-lift experiments — when you need them, synthetic control assumptions, and pre-test diagnostics
- [Switchback Experiments](https://confidence.spotify.com/rfp/rfp-switchback): What to ask about switchback designs for marketplace and network-effect settings where standard A/B tests break down

## Glossary

Confidence maintains a glossary of 131 experimentation, statistics, and feature management terms. Each term has a full article at the linked URL, and a plain-markdown version at the same URL with `.md` appended.

For the full glossary index, see: [Confidence Glossary](https://confidence.spotify.com/glossary)

### Core experimentation

- [A/A Testing](https://confidence.spotify.com/glossary/aa-testing): An A/A test is a randomized experiment where both groups receive the identical experience.
- [A/B Testing](https://confidence.spotify.com/glossary/ab-testing): An A/B test is a randomized controlled experiment that splits users into two groups: one sees the current experience (control), the other sees a changed version (treatment).
- [A/B/n Testing](https://confidence.spotify.com/glossary/abn-testing): An A/B/n test is a randomized experiment that compares more than two variants simultaneously: one control and two or more treatments.
- [Average Treatment Effect (ATE)](https://confidence.spotify.com/glossary/average-treatment-effect): The average treatment effect (ATE) is the mean difference in outcomes between the treatment group and the control group, averaged across the entire experimental population.
- [Control Group](https://confidence.spotify.com/glossary/control-group): The control group is the set of users in an experiment who see the unchanged, current experience.
- [Experiment Bandwidth](https://confidence.spotify.com/glossary/experiment-bandwidth): Experiment bandwidth is an organization's capacity to run concurrent experiments.
- [Experiment Design](https://confidence.spotify.com/glossary/experiment-design): Experiment design is the plan for how an experiment will be structured, sized, and analyzed before it begins.
- [Holdout Group](https://confidence.spotify.com/glossary/holdout-group): A holdout group is a segment of users permanently excluded from receiving a feature or set of features, maintained over time to measure cumulative long-term impact.
- [Hypothesis](https://confidence.spotify.com/glossary/hypothesis): A hypothesis is a testable prediction about the effect of a specific product change on a specific metric.
- [Interleaving](https://confidence.spotify.com/glossary/interleaving): Interleaving is an experiment technique where results from treatment and control are mixed within a single user session, rather than assigning each user entirely to one group.
- [Multivariate Testing](https://confidence.spotify.com/glossary/multivariate-testing): A multivariate test (MVT) is a randomized experiment that changes multiple variables simultaneously and measures both the individual effect of each variable and the interaction effects between them.
- [Mutually Exclusive Experiments](https://confidence.spotify.com/glossary/mutually-exclusive-experiments): Mutually exclusive experiments are experiments designed so that no user participates in more than one at the same time.
- [Null Hypothesis](https://confidence.spotify.com/glossary/null-hypothesis): The null hypothesis is the default assumption in a statistical test that there is no difference between the treatment and control groups.
- [Online Controlled Experiment](https://confidence.spotify.com/glossary/online-controlled-experiment): An online controlled experiment is the formal term for an A/B test conducted in a live digital product.
- [Product Builder](https://confidence.spotify.com/glossary/product-builder): A product builder is anyone who builds and ships product: engineers, product managers, designers, data scientists, and the increasingly blended roles between them.
- [Product Experimentation](https://confidence.spotify.com/glossary/product-experimentation): Product experimentation is the practice of using controlled experiments to validate product changes before full rollout.
- [Product Platform](https://confidence.spotify.com/glossary/product-platform): A product platform is the shared infrastructure and tooling that enables product teams to build, test, and ship features systematically.
- [Randomized Controlled Trial](https://confidence.spotify.com/glossary/randomized-controlled-trial): A randomized controlled trial (RCT) is an experimental design where participants are randomly assigned to a treatment group or a control group, and outcomes are compared between the groups to measure causal effects.
- [Scientific Product Development](https://confidence.spotify.com/glossary/scientific-product-development): Scientific product development is a product development approach that treats every product change as a hypothesis to be tested with experimental evidence before it ships.
- [Split Testing](https://confidence.spotify.com/glossary/split-testing): A split test is another name for an A/B test.
- [Target Audience](https://confidence.spotify.com/glossary/target-audience): A target audience is the subset of users eligible for an experiment, defined by targeting rules that filter on user attributes like country, platform, subscription tier, account age, or behavioral signals.
- [Treatment Effect](https://confidence.spotify.com/glossary/treatment-effect): A treatment effect is the measured difference in a metric between the treatment group (users who see a change) and the control group (users who see the current experience).
- [Treatment Group](https://confidence.spotify.com/glossary/treatment-group): The treatment group is the set of users in an experiment who see the changed experience.

### Statistical methods

- [Bayesian A/B Testing](https://confidence.spotify.com/glossary/bayesian-ab-testing): Bayesian A/B testing is an approach to experiment analysis that starts with a prior belief about the treatment effect and updates that belief using observed data, producing a posterior distribution.
- [Binary Metric](https://confidence.spotify.com/glossary/binary-metric): A binary metric is a metric that takes one of two values for each user in an experiment: 1 (the event happened) or 0 (it did not).
- [Causal Inference](https://confidence.spotify.com/glossary/causal-inference): Causal inference is the set of statistical methods used to determine whether a change actually caused an observed effect, rather than merely being correlated with it.
- [Confidence Interval](https://confidence.spotify.com/glossary/confidence-interval): A confidence interval is a range of values that, at a given confidence level, is expected to contain the true treatment effect.
- [CUPED](https://confidence.spotify.com/glossary/cuped): CUPED (Controlled-experiment Using Pre-Existing Data) is a variance reduction method that uses data from before an experiment started to remove predictable noise from metric estimates, producing tighter confidence intervals without requiring additional traffic.
- [Difference-in-Means Estimator](https://confidence.spotify.com/glossary/difference-in-means-estimator): The difference-in-means estimator is the simplest and most common estimator of the treatment effect in an A/B test.
- [Effect Size](https://confidence.spotify.com/glossary/effect-size): Effect size is the magnitude of the difference in a metric between treatment and control groups.
- [False Negative Rate (Type II Error)](https://confidence.spotify.com/glossary/false-negative-rate): The false negative rate, also called the Type II error rate or beta, is the probability of failing to detect a real treatment effect.
- [False Positive Rate (Type I Error)](https://confidence.spotify.com/glossary/false-positive-rate): The false positive rate, also called the Type I error rate, is the probability of concluding that a treatment had an effect when it actually did not.
- [Frequentist A/B Testing](https://confidence.spotify.com/glossary/frequentist-ab-testing): Frequentist A/B testing is the classical approach to experiment analysis that evaluates results using p-values and confidence intervals.
- [Metric Capping](https://confidence.spotify.com/glossary/metric-capping): Metric capping (also called winsorization) is a variance reduction technique that clips extreme metric values at a chosen threshold, reducing the outsized influence of outliers on experiment results.
- [Minimum Detectable Effect (MDE)](https://confidence.spotify.com/glossary/minimum-detectable-effect): The minimum detectable effect (MDE) is the smallest treatment effect an experiment is designed to reliably detect at a given significance level and power.
- [P-value](https://confidence.spotify.com/glossary/p-value): A p-value is the probability of observing a result at least as extreme as the one measured, assuming the null hypothesis is true (that is, assuming the change had no real effect).
- [Sample Size](https://confidence.spotify.com/glossary/sample-size): Sample size is the number of experimental units (typically users) needed in an A/B test to detect a given effect with a specified level of confidence and power.
- [Sampling Distribution](https://confidence.spotify.com/glossary/sampling-distribution): The sampling distribution is the probability distribution of a statistic (like a sample mean or a difference in means) computed across all possible random samples of a given size from a population.
- [Signal-to-Noise Ratio](https://confidence.spotify.com/glossary/signal-to-noise-ratio): The signal-to-noise ratio (SNR) in A/B testing is the ratio of the treatment effect (the signal) to the variability of the metric being measured (the noise).
- [Significance Level (Alpha)](https://confidence.spotify.com/glossary/significance-level): The significance level, commonly called alpha, is the maximum false positive rate you accept when declaring a result statistically significant.
- [Statistical Power](https://confidence.spotify.com/glossary/statistical-power): Statistical power is the probability that an experiment will detect a real effect when one exists.
- [Statistical Significance](https://confidence.spotify.com/glossary/statistical-significance): Statistical significance is the determination that an observed difference between experiment groups is unlikely to have occurred by chance alone.
- [Variance](https://confidence.spotify.com/glossary/variance): Variance is a measure of how much a metric's values spread around their mean.
- [Variance Reduction](https://confidence.spotify.com/glossary/variance-reduction): Variance reduction is a set of statistical techniques that tighten the confidence intervals of an A/B test without requiring more traffic.
- [Z-Test](https://confidence.spotify.com/glossary/z-test): A z-test is a hypothesis test that uses the standard normal distribution to determine whether the observed difference between two groups is statistically significant.

### Sequential testing

- [Alpha Spending](https://confidence.spotify.com/glossary/alpha-spending): Alpha spending is the method of distributing a fixed significance budget (alpha, typically 5%) across multiple interim analyses in a group sequential test.
- [Always-Valid Inference](https://confidence.spotify.com/glossary/always-valid-inference): Always-valid inference (AVI) is a class of sequential testing methods that construct confidence intervals remaining valid at any stopping time, without requiring the experimenter to pre-plan when or how often to look at results.
- [Fixed-Power Design](https://confidence.spotify.com/glossary/fixed-power-design): A fixed-power design is a sequential experiment plan where the stopping rule is based on achieving a pre-specified level of statistical power rather than on observing a statistically significant result.
- [Group Sequential Test](https://confidence.spotify.com/glossary/group-sequential-test): A group sequential test (GST) is a sequential testing method that pre-plans a fixed number of interim analyses at specific points during an experiment, using an alpha spending function to distribute the significance budget.
- [Information Fraction](https://confidence.spotify.com/glossary/information-fraction): The information fraction is the proportion of the total planned statistical information that has been observed so far in a sequential experiment.
- [Optional Stopping](https://confidence.spotify.com/glossary/optional-stopping): Optional stopping is the practice of ending an experiment based on observed results rather than a pre-determined stopping rule.
- [Peeking Problem](https://confidence.spotify.com/glossary/peeking-problem): The peeking problem is the inflation of false positive rates that occurs when experimenters check statistical results before the planned sample size has been reached and stop the experiment early based on those results.
- [Sequential Testing](https://confidence.spotify.com/glossary/sequential-testing): Sequential testing is a statistical framework that allows experimenters to make valid decisions at multiple analysis points during an experiment, rather than waiting for a single final evaluation.

### Multiple testing

- [Benjamini-Hochberg Correction](https://confidence.spotify.com/glossary/benjamini-hochberg-correction): The Benjamini-Hochberg (BH) correction is a multiple testing procedure that controls the false discovery rate (FDR): the expected proportion of false positives among all results declared significant.
- [Bonferroni Correction](https://confidence.spotify.com/glossary/bonferroni-correction): The Bonferroni correction adjusts significance thresholds for multiple testing by dividing the target alpha by the number of tests.
- [Correction Family](https://confidence.spotify.com/glossary/correction-family): A correction family is the set of hypothesis tests grouped together for a multiple testing adjustment.
- [False Discovery Rate](https://confidence.spotify.com/glossary/false-discovery-rate): False discovery rate (FDR) is the expected proportion of false positives among all results declared statistically significant.
- [Family-Wise Error Rate](https://confidence.spotify.com/glossary/family-wise-error-rate): Family-wise error rate (FWER) is the probability of making at least one false positive across a set of hypothesis tests.
- [Holm Correction](https://confidence.spotify.com/glossary/holm-correction): The Holm correction (also called Holm-Bonferroni) is a step-down multiple testing procedure that controls the family-wise error rate (FWER) while being uniformly more powerful than the Bonferroni correction.
- [Hommel Correction](https://confidence.spotify.com/glossary/hommel-correction): The Hommel correction is a multiple testing procedure that controls the family-wise error rate (FWER) while being more powerful than both the Bonferroni correction and the Holm correction.
- [Multiple Testing Correction](https://confidence.spotify.com/glossary/multiple-testing-correction): A multiple testing correction is an adjustment to significance thresholds that accounts for evaluating more than one hypothesis in the same experiment.

### Metrics

- [Conversion Rate](https://confidence.spotify.com/glossary/conversion-rate): Conversion rate is the fraction of users who complete a desired action out of those who had the opportunity to complete it.
- [Daily Active Users (DAU)](https://confidence.spotify.com/glossary/daily-active-users): Daily active users (DAU) is the count of unique users who engage with a product within a single calendar day.
- [Guardrail Metric](https://confidence.spotify.com/glossary/guardrail-metric): A guardrail metric is a metric monitored during an experiment to ensure the change does not cause unintended harm, even when the success metric improves.
- [Inferiority Test](https://confidence.spotify.com/glossary/inferiority-test): An inferiority test checks whether a treatment is worse than control by more than a specified margin on a guardrail metric.
- [Longitudinal Guardrails](https://confidence.spotify.com/glossary/longitudinal-guardrails): Longitudinal guardrails track guardrail metrics across many experiments over time to detect slow, cumulative harm that no individual experiment would flag.
- [Metric Drift](https://confidence.spotify.com/glossary/metric-drift): Metric drift is a gradual, non-experiment-related change in a metric's baseline value over time.
- [Metric Interaction Effects](https://confidence.spotify.com/glossary/metric-interaction-effects): Metric interaction effects occur when the effect of one experiment on a metric depends on whether another experiment is also running.
- [Metric Sensitivity](https://confidence.spotify.com/glossary/metric-sensitivity): Metric sensitivity is how responsive a metric is to real product changes.
- [Monthly Active Users (MAU)](https://confidence.spotify.com/glossary/monthly-active-users): Monthly active users (MAU) is the count of unique users who engage with a product within a 30-day (or calendar month) window.
- [Non-inferiority Margin](https://confidence.spotify.com/glossary/non-inferiority-margin): A non-inferiority margin (NIM) is the maximum amount of deterioration in a guardrail metric that a team is willing to accept in exchange for a gain on their success metric.
- [Non-inferiority Test](https://confidence.spotify.com/glossary/non-inferiority-test): A non-inferiority test confirms that a treatment is not meaningfully worse than control on a guardrail metric.
- [North Star Metric](https://confidence.spotify.com/glossary/north-star-metric): A north star metric is the single metric that best captures the value a product delivers to its users.
- [Primary Metric](https://confidence.spotify.com/glossary/primary-metric): A primary metric is the main metric used to decide whether to ship an experiment's treatment.
- [Proxy Metric](https://confidence.spotify.com/glossary/proxy-metric): A proxy metric is a measurable stand-in for a harder-to-measure outcome.
- [Secondary Metric](https://confidence.spotify.com/glossary/secondary-metric): A secondary metric is a supporting metric that provides additional context about an experiment's impact beyond the primary metric.
- [Ship Rate](https://confidence.spotify.com/glossary/ship-rate): Ship rate is the proportion of experiments whose results lead to shipping the treatment to all users.
- [Success Metric](https://confidence.spotify.com/glossary/success-metric): A success metric is the primary metric an experiment is designed to move.
- [Weekly Active Users (WAU)](https://confidence.spotify.com/glossary/weekly-active-users): Weekly active users (WAU) is the count of unique users who engage with a product within a rolling or fixed 7-day window.
- [Win Rate](https://confidence.spotify.com/glossary/win-rate): Win rate is the proportion of experiments that produce a statistically significant positive result on their primary success metric.

### Feature flags and rollouts

- [Auto-Rollback](https://confidence.spotify.com/glossary/auto-rollback): Auto-rollback is an automatic rollback triggered when a guardrail metric violates a predefined threshold during a rollout.
- [Bucket Hashing](https://confidence.spotify.com/glossary/bucket-hashing): Bucket hashing is the mechanism that maps a user into a numbered bucket, which then determines their variant in a feature flag or experiment.
- [Canary Release](https://confidence.spotify.com/glossary/canary-release): A canary release exposes a change to a tiny fraction of traffic first, typically 1% or less, to detect severe problems before wider rollout.
- [Deterministic Assignment](https://confidence.spotify.com/glossary/deterministic-assignment): Deterministic assignment is a method of assigning users to experiment variants by hashing a stable identifier (typically the user ID) combined with a salt, so that the same user always maps to the same variant.
- [Dynamic Config](https://confidence.spotify.com/glossary/dynamic-config): A dynamic config is a feature flag that carries structured data (JSON objects, typed fields) instead of a simple boolean.
- [Edge Resolvers](https://confidence.spotify.com/glossary/edge-resolvers): Edge resolvers evaluate feature flags at the CDN or edge compute layer, before a request reaches the origin server.
- [Experimentation Platform](https://confidence.spotify.com/glossary/experimentation-platform): An experimentation platform is the end-to-end system that powers controlled experiments at scale: feature flags for assignment, metric pipelines for measurement, a statistical engine for analysis, and dashboards for decisions.
- [Feature Flag](https://confidence.spotify.com/glossary/feature-flag): A feature flag is a runtime switch that controls whether a feature is active for a given user, without deploying new code.
- [Feature Toggle](https://confidence.spotify.com/glossary/feature-toggle): A feature toggle is a runtime switch that controls whether a feature is active for a given user, without deploying new code.
- [Flag Rule](https://confidence.spotify.com/glossary/flag-rule): A flag rule is a conditional expression that determines which users see which variant of a feature flag.
- [Holdback](https://confidence.spotify.com/glossary/holdback): A holdback is a subset of users intentionally kept on the old experience after a feature has shipped to 100% of everyone else.
- [Kill Switch](https://confidence.spotify.com/glossary/kill-switch): A kill switch is a feature flag designed specifically for instant emergency deactivation of a feature.
- [Local Evaluation](https://confidence.spotify.com/glossary/local-evaluation): Local evaluation resolves feature flags on the client or server without making a network call to a central service at evaluation time.
- [OpenFeature](https://confidence.spotify.com/glossary/openfeature): OpenFeature is a CNCF (Cloud Native Computing Foundation) open standard that defines a vendor-agnostic API for feature flag management.
- [Overrides](https://confidence.spotify.com/glossary/overrides): Overrides manually force a specific user or group into a particular feature flag variant, bypassing the normal assignment logic.
- [Phased Rollout](https://confidence.spotify.com/glossary/phased-rollout): A phased rollout is a progressive rollout organized into discrete stages, each with a predefined percentage and explicit criteria that must be met before advancing to the next stage.
- [Progressive Rollout](https://confidence.spotify.com/glossary/progressive-rollout): A progressive rollout is the practice of gradually increasing the percentage of users exposed to a feature over time, rather than releasing it to everyone at once.
- [Rollback](https://confidence.spotify.com/glossary/rollback): A rollback reverts a feature to its previous state when problems are detected during a release.
- [Rollout](https://confidence.spotify.com/glossary/rollout): A rollout is the process of releasing a feature to users in controlled stages using feature flags.
- [Sticky Assignments](https://confidence.spotify.com/glossary/sticky-assignments): Sticky assignments ensure that a user who has been assigned to a variant in a feature flag or experiment continues to see that same variant across sessions, devices, and app restarts.
- [Targeting Conditions](https://confidence.spotify.com/glossary/targeting-conditions): Targeting conditions are the criteria a feature flag evaluates to decide which variant a user receives.
- [Warehouse-Native Experimentation](https://confidence.spotify.com/glossary/warehouse-native-experimentation): Warehouse-native experimentation is an architecture where experiment data (assignments, exposures, metric events, and analysis results) lives in the customer's own data warehouse rather than being copied into the vendor's infrastructure.

### Experiment analysis

- [Confounding Variables](https://confidence.spotify.com/glossary/confounding-variables): A confounding variable is a factor that influences both the treatment assignment and the outcome being measured, creating a spurious association that can be mistaken for a causal effect.
- [Counterfactual Logging](https://confidence.spotify.com/glossary/counterfactual-logging): Counterfactual logging is the practice of recording what a system would have shown a user under an alternative policy or variant, alongside what was actually shown.
- [Dilution](https://confidence.spotify.com/glossary/dilution): Dilution is the weakening of an observed treatment effect that occurs when users who were never exposed to the changed feature are included in the experiment analysis.
- [Estimand](https://confidence.spotify.com/glossary/estimand): An estimand is the precise quantity an experiment is designed to estimate.
- [Exposure Filters](https://confidence.spotify.com/glossary/exposure-filters): Exposure filters are criteria applied during experiment analysis to include or exclude users based on their exposure to the treatment.
- [Exposure Logging](https://confidence.spotify.com/glossary/exposure-logging): Exposure logging is the practice of recording exactly when and whether each user was actually exposed to a specific experiment variant.
- [Garden of Forking Paths](https://confidence.spotify.com/glossary/garden-of-forking-paths): The garden of forking paths refers to the many implicit analytical choices a researcher or analyst makes during an experiment's lifecycle that inflate the risk of false positives.
- [Multi-Armed Bandit](https://confidence.spotify.com/glossary/multi-armed-bandit): A multi-armed bandit is an adaptive experiment design that shifts traffic allocation toward better-performing variants during the experiment, rather than splitting traffic evenly for the entire duration.
- [Observational Bias](https://confidence.spotify.com/glossary/observational-bias): Observational bias is systematic error introduced when the data collection or analysis process produces results that consistently differ from the truth.
- [Sample Ratio Mismatch](https://confidence.spotify.com/glossary/sample-ratio-mismatch): A sample ratio mismatch (SRM) occurs when the observed number of users in each experiment group differs from the intended allocation ratio by more than chance alone would explain.
- [Segment Analysis](https://confidence.spotify.com/glossary/segment-analysis): Segment analysis breaks down experiment results by user subgroups to detect heterogeneous treatment effects: cases where the change helps some users, hurts others, or has no effect on a particular segment.
- [Simpson's Paradox](https://confidence.spotify.com/glossary/simpsons-paradox): Simpson's paradox occurs when a trend that appears in several subgroups reverses or disappears when the subgroups are combined.
- [Trigger Analysis](https://confidence.spotify.com/glossary/trigger-analysis): Trigger analysis is an experiment analysis technique that restricts the evaluation to users who actually encountered the changed feature, rather than analyzing every user assigned to the experiment.
- [Unbiased Estimator](https://confidence.spotify.com/glossary/unbiased-estimator): An unbiased estimator is a statistical estimator whose expected value equals the true parameter it is estimating.

### Culture and organization

- [Build-Measure-Learn](https://confidence.spotify.com/glossary/build-measure-learn): Build-measure-learn is the iterative product development loop introduced by Eric Ries in The Lean Startup.
- [Continuous Discovery](https://confidence.spotify.com/glossary/continuous-discovery): Continuous discovery is an ongoing cycle of learning from users and experiments, where research, hypothesis formation, and validation happen continuously alongside product development.
- [Cumulative Holdback Evaluation](https://confidence.spotify.com/glossary/cumulative-holdback-evaluation): Cumulative holdback evaluation is a method for measuring the aggregate impact of many shipped features by maintaining a long-running holdout group that does not receive new features.
- [Dogfooding](https://confidence.spotify.com/glossary/dogfooding): Dogfooding is the practice of using your own product internally before releasing it to customers.
- [Experiment Coordination](https://confidence.spotify.com/glossary/experiment-coordination): Experiment coordination is the practice of managing interactions, priorities, and resource allocation across concurrent experiments.
- [Experimentation Culture](https://confidence.spotify.com/glossary/experimentation-culture): Experimentation culture is the organizational norm of testing product ideas with data before committing to them.
- [Experimentation Maturity Model](https://confidence.spotify.com/glossary/experimentation-maturity-model): An experimentation maturity model is a framework for assessing how advanced an organization's experimentation practice is across dimensions like tooling, statistical rigor, and organizational adoption.
- [Experimentation Theatre](https://confidence.spotify.com/glossary/experimentation-theatre): Experimentation theatre is the practice of running experiments without the rigor or organizational commitment to act on results.
- [Hypothesis-Driven Development](https://confidence.spotify.com/glossary/hypothesis-driven-development): Hypothesis-driven development is a product development approach where each change is framed as a testable hypothesis before it is built.
- [Multi-Metric Decision Making](https://confidence.spotify.com/glossary/multi-metric-decision-making): Multi-metric decision making is the practice of evaluating experiment results across multiple metrics simultaneously rather than basing ship decisions on a single success metric.
- [Product Loop](https://confidence.spotify.com/glossary/product-loop): The product loop is the recurring cycle of insight, hypothesis, experiment, decision, and insight that drives how a product improves over time.

### Conversion and optimization

- [Client-Side Testing](https://confidence.spotify.com/glossary/client-side-testing): Client-side testing is the practice of running experiments in the browser using injected JavaScript that modifies the page after it loads.
- [Conversion Rate Optimization](https://confidence.spotify.com/glossary/conversion-rate-optimization): Conversion rate optimization (CRO) is the practice of systematically improving the percentage of users who complete a desired action.
- [Server-Side Testing](https://confidence.spotify.com/glossary/server-side-testing): Server-side testing is the practice of running experiments in backend code rather than in the browser.
- [Website Optimization](https://confidence.spotify.com/glossary/website-optimization): Website optimization is the practice of improving a website's performance, usability, and conversion rates through systematic testing and iteration.