> ## Documentation Index
> Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Get Started with BigQuery

> Configure read and write connections to BigQuery.

export const NotForSpotify = () => <Note>
    <strong>This page doesn't apply at Spotify.</strong> Use the Spotify
    documentation in the sidebar.
  </Note>;

{user?.groups?.includes("spotify") && <NotForSpotify />}

<img src="https://mintcdn.com/confidence-7c0fec1b/jMySNzOX-h8w-Z-A/images/bigquery-logo.png?fit=max&auto=format&n=jMySNzOX-h8w-Z-A&q=85&s=3b8689155f8b1c591f8599defc02c09f" alt="BigQuery logo" width="1100" height="613" data-path="images/bigquery-logo.png" />

## Introduction

<Note>
  If you followed the setup guide on the **Admin** page, your
  setup is already complete. You can skip directly to [what's next](#whats-next).
</Note>

<Tip>
  You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:

  ```bash theme={null}
  npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-bigquery
  ```

  The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.
</Tip>

This tutorial helps you configure Confidence to:

1. Run queries in BigQuery to compute exposure and metrics.
2. Store assignment data in BigQuery.

Step 2 is optional if you already have assignment data in BigQuery. For
example, if you are using a feature flagging solution other than Confidence
Flags.

This document targets the following audiences:

* Administrators who want to set up Confidence for their organization

<Note>
  In this tutorial, you create a service account in GCP that Confidence impersonates.
  The service account used by Confidence is `account-<your-account-id>@spotify-confidence.iam.gserviceaccount.com`.
  You can name your service account anything you want.
  For clarity, the tutorial assumes you create a service account called `confidence`.
  You can read more about service account impersonation in the <a href="https://cloud.google.com/iam/docs/service-account-impersonation" target="_blank">Google Cloud documentation</a>.
</Note>

## Before You Begin

* You need to have a Confidence account.
* You need to have a Google Cloud Platform account.
* You need to have permissions to create service accounts.
* You need to have the following APIs enabled in your Google Cloud Platform
  project:
  * BigQuery API
  * Cloud Resource Manager API

## Terraform Scripts

<a href="https://confidence-cdn.spotifycdn.com/terraform/bq_terraform.tar.gz" download>Terraform scripts are available</a> to create the service account, set up permissions, and create the required datasets.

If you are using Terraform, then after you have applied the Terraform configuration, you can skip directly to [Step 1c](#step-1c-configure-a-metrics-data-warehouse).

## Step 1: Prepare GCP and Create Data Warehouse Connection

In this step, you:

* create a service account in GCP that Confidence impersonates
* create a dataset for storing the tables that Confidence generates
* configure a metrics data warehouse connection

### Step 1a: Create Service Account

For Confidence to access BigQuery, you need to create a service account that
has permissions to access BigQuery. You also need to allow Confidence to
impersonate the service account. Follow these steps:

<Steps>
  <Step title="Go to Google Cloud Platform Console">
    Go to the <a href="https://console.cloud.google.com/" target="_blank">Google Cloud Platform Console</a> and select the project you want to use for Confidence.
  </Step>

  <Step title="Navigate to Service Accounts">
    Go to <a href="https://console.cloud.google.com/iam-admin/serviceaccounts" target="_blank">IAM & Admin > Service Accounts</a>.
  </Step>

  <Step title="Click + CREATE SERVICE ACCOUNT" />

  <Step title="Enter service account details">
    Enter a name for the service account, for example, `confidence`. Make note of the service account email address. You can click copy-to-clipboard icon to copy it. Click **CREATE AND CONTINUE**.
  </Step>

  <Step title="Grant roles to the account">
    Grant the following roles to the account:

    * **BigQuery Job User**
    * **BigQuery Data Viewer**

    Click **CONTINUE**.
  </Step>

  <Step title="Click DONE" />
</Steps>

#### Grant Confidence Access to Impersonate the Service Account

Confidence needs to be able to impersonate the service account you just created
to be able to access BigQuery. You do this by granting one of Confidence's own
service accounts the "Workload Identity User" role, called the principal account:

```text theme={null}
account-<your-account-id>@spotify-confidence.iam.gserviceaccount.com
```

To grant the principal account access to impersonate the service account, follow
these steps:

<Steps>
  <Step title="Open the service account details">
    Click on the service account you just created to go to the service account details page.
  </Step>

  <Step title="Select the PRINCIPALS WITH ACCESS tab" />

  <Step title="Click GRANT ACCESS" />

  <Step title="Enter the principal account">
    Enter `account-<your-account-id>@spotify-confidence.iam.gserviceaccount.com` as principal.
  </Step>

  <Step title="Assign the Workload Identity User role" />

  <Step title="Click SAVE" />
</Steps>

### Step 1b: Create BigQuery Datasets to Store Metrics and Exposure

Confidence needs a dataset in BigQuery where it can store intermediate metrics
and exposure calculations. For Confidence to access the dataset,
you need to grant the principal service account access to the dataset.

```text theme={null}
account-<your-account-id>@spotify-confidence.iam.gserviceaccount.com
```

To create the dataset, follow these steps:

<Steps>
  <Step title="Go to Google Cloud Platform Console">
    Go to the <a href="https://console.cloud.google.com/" target="_blank">Google Cloud Platform Console</a> and select the project you want to use for Confidence.
  </Step>

  <Step title="Create a new dataset">
    In <a href="https://console.cloud.google.com/bigquery" target="_blank">BigQuery</a>, click **⋮ > CREATE DATASET** next to the name of the project you want to create it in.
  </Step>

  <Step title="Enter dataset name">
    Enter a name for the dataset. For example, `confidence_metrics`.
  </Step>

  <Step title="Select the dataset location">
    Select the location where you want to store the dataset.

    <Note>
      The dataset location must be the same as the region of your Confidence account. The recommendation is to select a multi-region location for your dataset, like `EU` or `US`.
    </Note>
  </Step>

  <Step title="Click CREATE DATASET" />

  <Step title="Click GO TO DATASET" />

  <Step title="Click SHARE > Manage Permissions" />

  <Step title="Click ADD PRINCIPAL" />

  <Step title="Grant permissions">
    Enter the email address of the `confidence` service account you created and assign the role **BigQuery Data Owner**.
  </Step>

  <Step title="Click SAVE" />
</Steps>

### Step 1c: Configure a Metrics Data Warehouse

With the completion of steps 1a and 1b, you have now created a service account and
a dataset in BigQuery. You can now configure Confidence to use BigQuery to
compute metrics. To do so, follow these steps:

<Steps>
  <Step title="Go to the Confidence app" />

  <Step title="Navigate to Metrics Data Warehouse settings">
    On the bottom of the left sidebar, select **Admin > Connections > Metrics Data Warehouse**.
  </Step>

  <Step title="Select BigQuery" />

  <Step title="Enter connection details">
    Enter the GCP Project ID, your metrics dataset (`confidence_metrics`) and the email address of the `confidence` service account you created in step 1a.
  </Step>

  <Step title="Click SAVE" />
</Steps>

When you click save, Confidence tries to connect to BigQuery using the
service account you created in step 1a. If you have misconfigured anything,
you should see an error message.

**🎉 Well done! Now you've configured Confidence to compute metrics using BigQuery.**

If you plan to use Confidence Flags, you also need to configure Confidence to
store assignment data in BigQuery. To do so, follow the next steps outlined
below.

## Step 2: Configure Confidence to Write and Read Assignment Data

In this step, you:

* create a dataset in BigQuery that Confidence can write assignment data to
* configure a flag applied connector so that Confidence is able to write assignment data
* configure an assignment table so that Confidence is able to read assignment data

### Step 2a: Create BigQuery Datasets to Store Assignment Data

If you want to use Confidence Flags to run experiments, you need to complete
a few more steps to configure Confidence to store assignment data in BigQuery
and an assignment table that reads from the BigQuery table.

Confidence needs a dataset in BigQuery where it can store assignment data. For
Confidence to access the dataset, you need to grant the principal
service account access to the dataset.

```text theme={null}
account-<your-account-id>@spotify-confidence.iam.gserviceaccount.com
```

To create the dataset, follow these steps:

<Steps>
  <Step title="Go to Google Cloud Platform Console">
    Go to the <a href="https://console.cloud.google.com/" target="_blank">Google Cloud Platform Console</a> and select the project you want to use for Confidence.
  </Step>

  <Step title="Create a new dataset">
    Go to <a href="https://console.cloud.google.com/bigquery" target="_blank">BigQuery</a> and click **⋮ > CREATE DATASET** next to the name of the project you want to create it in.
  </Step>

  <Step title="Enter dataset name">
    Enter a name for the dataset, for example, `confidence_flag_applied`.
  </Step>

  <Step title="Select the dataset location">
    Select the location where you want to store the dataset.

    <Note>
      The dataset location must be the same as the region of your Confidence account. The recommendation is to select a multi-region location for your dataset, like `EU` or `US`.
    </Note>
  </Step>

  <Step title="Click CREATE DATASET" />

  <Step title="Click GO TO DATASET" />

  <Step title="Click SHARE and then Manage Permissions" />

  <Step title="Click ADD PRINCIPAL" />

  <Step title="Grant permissions">
    Enter the email address of the `confidence` service account you created and add the role **BigQuery Data Owner**.
  </Step>

  <Step title="Click SAVE" />
</Steps>

### Step 2b: Configure a Flag Applied Connector

For Confidence to be able to store assignment data in BigQuery, you need to set
up a connector between Confidence and BigQuery.

<Note>
  Assignment data is information on which users were assigned to which variants
  in the experiments you run. Assignment data goes into exposure calculations.
  Metrics use exposure to calculate results in your tests.
</Note>

This connector is a **"Flag Applied"** connector.
The connector is the part responsible for writing assignment data to BigQuery that Confidence Metrics
can later read.

<Steps>
  <Step title="Go to the Confidence App" />

  <Step title="Navigate to Flag applied connections">
    On the bottom of the left sidebar, select **Admin > Connections > Flag applied connections**.
  </Step>

  <Step title="Select BigQuery" />

  <Step title="Click Next" />

  <Step title="Enter connection details">
    Enter the Project ID, the dataset (for example, `confidence_flag_applied`) you created in step 2a and the email address of the `confidence` service account you created in step 1a.
  </Step>

  <Step title="Enter the table name">
    Enter the name of the table (for example, `flag_applied`) to write assignment data to. Confidence creates the table with the right schema for you.
  </Step>

  <Step title="Click Create" />
</Steps>

When you click save, Confidence tries to connect to BigQuery using the
service account you created in step 1a. If you have misconfigured anything,
you should see an error message.

### Step 2c: Configure an Assignment Table

For Confidence to use the stored assignment table, you need to set
up an assignment table that reads from the BigQuery table. You first need to create
an entity, which represents the thing you're experimenting on, like your users.
To do so, follow these steps:

<Steps>
  <Step title="Go to the Confidence App" />

  <Step title="Navigate to the BigQuery connection">
    On the bottom of the left sidebar, select **Admin > Connections > Flag applied connections** and then the BigQuery connection you created.
  </Step>

  <Step title="Click Create in the Assignment table section" />

  <Step title="Enter assignment table name">
    Enter a name for the assignment table, such as `flag_applied`. This name should typically match the name you used in step 2b. Confidence can then read assignments from the destination table of your flag assignments.
  </Step>

  <Step title="Create or select an entity">
    Create a new entity or select an existing entity. Entities are the things you're experimenting on, like your users. Enter `User` and specify the data type of the identifier that identifies the entity. For example, if you have a UUID that identifies your users, your primary key type is a **String**.
  </Step>

  <Step title="Click Create" />
</Steps>

**🎉 Well done! You are all set up and ready to go.**

## Step 3: Grant View Access to Metrics Data Sources

Confidence uses fact tables to describe the data that you create metrics from.
To ensure that Confidence can access data from the data sources you use for fact tables, you
need to grant the role **BigQuery Data Viewer** to the service account named `confidence` that you have created.
Grant this role for the datasets and tables you want Confidence to have access to.

<Steps>
  <Step title="Go to Google Cloud Platform Console">
    Go to the <a href="https://console.cloud.google.com/" target="_blank">Google Cloud Platform Console</a> and select the project that the data you want to grant Confidence access to is in.
  </Step>

  <Step title="Select the dataset or table">
    Go to <a href="https://console.cloud.google.com/bigquery" target="_blank">BigQuery</a> and select the dataset. Optionally select the table.
  </Step>

  <Step title="Open permissions dialog">
    If you selected a dataset, click **SHARING > Manage Permissions**. If you selected a table, click **SHARE**.
  </Step>

  <Step title="Click ADD PRINCIPAL" />

  <Step title="Grant permissions">
    Enter the email address of the `confidence` service account you created and assign the role **BigQuery Data Viewer**.
  </Step>

  <Step title="Click SAVE" />
</Steps>

## What's Next?

The next step is to [create a fact table](/docs/metrics/fact-tables), and a
[metric](/docs/metrics/introduction). For an overview, see the [metric introduction](/docs/metrics/introduction)
page, and the [metrics quickstart](/docs/quickstarts/configure-metric).

## Related Resources

<CardGroup cols={2}>
  <Card title="Metrics Introduction" href="/docs/metrics/introduction">
    Overview of metrics in Confidence
  </Card>

  <Card title="Fact Tables" href="/docs/metrics/fact-tables">
    Configure fact tables for metrics
  </Card>

  <Card title="Assignment Tables" href="/docs/metrics/assignment-tables">
    Set up assignment tracking
  </Card>

  <Card title="Configure Metric Quickstart" href="/docs/quickstarts/configure-metric">
    Create your first metric
  </Card>
</CardGroup>