> ## Documentation Index
> Fetch the complete documentation index at: https://confidence.spotify.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Get started with Databricks

> Configure Confidence to use Databricks with supported cloud staging providers.

<Tip>
  You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:

  ```bash theme={null}
  npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-databricks
  ```

  The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.
</Tip>

This guide explains how to configure Confidence to use Databricks for metrics, events, and flag assignment data. It covers the Databricks setup shared by all three use cases, then shows how to configure a supported staging provider for temporary assignment and event files.

This guide is for administrators who manage Confidence, Databricks, and the relevant cloud storage resources for their organization.

<Note>
  Staging storage is required only for Event and Flag Applied connectors. If your assignment data already exists in Databricks, for example because another feature-flagging platform writes it there, you can skip the Flag Applied connector setup and [create an assignment table](/docs/capabilities/metrics/create-assignment-table) that queries the existing data.

  If you are not configuring an Event or Flag Applied connector, you can also skip the staging storage setup.
</Note>

Confidence uses two separate identities:

| Identity | Used for | Configured in Confidence as |
| - | - | - |
| Databricks service principal | Authenticates to Databricks and runs SQL on a SQL warehouse | Host, Warehouse ID, Client ID, and Client Secret |
| Staging storage identity | Writes temporary files and lets Databricks read them during `COPY INTO` | The selected staging provider's storage and identity settings |

The Databricks service principal and the staging-provider identity are separate identities. You can reuse the same Databricks service principal and SQL warehouse across staging-provider configurations, but configure only one staging provider for each connector.

## Before you begin

* You need a Confidence account.
* You need access to a Databricks workspace with Unity Catalog enabled.
* You need permission to create or reuse a SQL warehouse, service principal, catalog/schema, and grants in Databricks.
* To configure an Event or Flag Applied connector, you need permission to set up S3 or ADLS Gen2 staging in AWS or Azure. Step 5 lists what you need.

## Step 1: Create or reuse a SQL warehouse

Confidence sends SQL statements to a Databricks SQL warehouse. You can create a new warehouse or reuse an existing warehouse in the workspace.

1. In Databricks, open **SQL warehouses**.
2. Create a warehouse, or select an existing warehouse that Confidence can use.
3. Copy the warehouse ID.
4. Open the warehouse **Connection details** and copy the server hostname.

Both connections use the same Databricks workspace host. The **Host** field uses a different format depending on the connection:

| Confidence connection | Value to enter |
| - | - |
| Event or Flag Applied | `https://` followed by the server hostname, such as `https://adb-1234567890123456.7.azuredatabricks.net` |
| Metrics Data Warehouse | The **Server hostname** from the warehouse connection details, such as `adb-1234567890123456.7.azuredatabricks.net` |

Use the SQL warehouse ID for **Warehouse ID**. The warehouse ID is not the JDBC HTTP path.

Keep the warehouse running, or configure its start-up and auto-stop settings so that Confidence can start it when it needs to run a query.

For more information, see the Databricks documentation for [SQL warehouse access control](https://docs.databricks.com/aws/en/security/auth/access-control/).

## Step 2: Create a Databricks service principal

Create a Databricks service principal for Confidence. This service principal is used for OAuth machine-to-machine authentication and for running SQL. It is separate from the identity used by the selected staging provider.

1. In the Databricks account console, open **Identity and access > Service principals**.
2. Create a service principal.
3. Assign the service principal to the Databricks workspace that contains the SQL warehouse.
4. Give it the **Databricks SQL access** entitlement.
5. Create an OAuth secret for the service principal. Save the service principal application ID, also called the OAuth client ID, and the generated OAuth secret securely.

The client ID is the persistent application ID of the service principal. The generated secret is the credential paired with that ID. Confidence uses these two values in the Databricks connection form; it does not require an OAuth access token or a personal access token.

See [Databricks OAuth machine-to-machine authentication](https://docs.databricks.com/aws/en/dev-tools/auth/) for the current Databricks account and workspace steps.

## Step 3: Grant the service principal access to the warehouse

The service principal must be allowed to run queries on the SQL warehouse.

1. Open the SQL warehouse in Databricks.
2. Open **Permissions**.
3. Add the Confidence service principal.
4. Grant **Can use**.

The **Can use** permission is required even when the service principal has access to the catalog and schema. The service principal does not need **Can manage** or **Can monitor** to run Confidence queries.

## Step 4: Create a catalog and schema

Choose the Unity Catalog catalog and schema where Confidence will create its tables. In the Confidence connection forms, enter the fully qualified schema as `catalog.schema`. For example:

```text theme={null}
main.confidence
```

Create the schema as a Databricks administrator or another principal with permission to create schemas:

```sql theme={null}
CREATE SCHEMA IF NOT EXISTS <catalog>.<confidence_schema>;
```

Replace the placeholders with your catalog and schema names. For example:

```sql theme={null}
CREATE SCHEMA IF NOT EXISTS main.confidence;
```

Grant the Databricks service principal the permissions Confidence needs:

```sql theme={null}
GRANT USE CATALOG ON CATALOG <catalog>
  TO `<databricks_service_principal_application_id>`;

GRANT USE SCHEMA ON SCHEMA <catalog>.<confidence_schema>
  TO `<databricks_service_principal_application_id>`;

GRANT CREATE TABLE ON SCHEMA <catalog>.<confidence_schema>
  TO `<databricks_service_principal_application_id>`;
```

Confidence creates its connector tables with `CREATE TABLE IF NOT EXISTS`. You do not need to grant the service principal permission to create schemas unless you want Confidence to create schemas on your behalf.

If a connector table already exists and is owned by another principal, grant the service principal permission to modify that table:

```sql theme={null}
GRANT MODIFY ON TABLE <catalog>.<confidence_schema>.<table>
  TO `<databricks_service_principal_application_id>`;
```

If you use the same service principal for the Metrics Data Warehouse, grant it read access to the tables that contain your metric facts and dimensions. Grant access to individual tables when possible:

```sql theme={null}
GRANT SELECT ON TABLE <catalog>.<metrics_schema>.<metric_table>
  TO `<databricks_service_principal_application_id>`;
```

You can grant access to all current or future tables in a schema when that is appropriate for your governance model. See the Databricks documentation for [Unity Catalog privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/permissions-concepts) and [`GRANT` syntax](https://docs.databricks.com/aws/en/sql/language-manual/security-grant).

## Step 5: Configure staging storage

This is the point where the setup differs by staging provider. The SQL warehouse, Databricks service principal, catalog, schema, and Unity Catalog permissions above are shared. Choose one supported staging provider for each Event or Flag Applied connector.

<Tabs>
  <Tab title="Amazon S3">
    Confidence writes Parquet files to S3 and then runs `COPY INTO` in Databricks. Confidence uses the IAM role to access the bucket and supplies short-lived credentials to Databricks for the load.

    ### Create or reuse an S3 bucket

    Create an S3 bucket, or choose an existing bucket. Select the AWS region where the bucket is located. The bucket must be in the region selected in the connection form.

    ### Configure the IAM role

    The role must trust the Confidence service account. The S3 form supports either web identity authentication or an IAM access key and secret.

    #### Recommended: web identity authentication

    1. In the AWS IAM console, create a role with a custom trust policy.
    2. Leave **Authenticate with credentials** unchecked in the Confidence form. The form shows the Confidence service account ID to use in the trust policy.
    3. Replace `<confidence_service_account_id>` in the following policy with that ID:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "accounts.google.com"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "accounts.google.com:sub": "<confidence_service_account_id>"
            }
          }
        }
      ]
    }
    ```

    #### Alternative: IAM user credentials

    If web identity authentication is not suitable for your AWS account, create an IAM user with an access key and secret. Configure the role trust policy to trust that user's ARN and grant the user `sts:AssumeRole` on the role. Enter the access key and secret by selecting **Authenticate with credentials** in the Confidence form.

    ### Grant the role access to the bucket

    Attach an inline or managed policy to the role. Replace `<s3_bucket_name>` with the bucket name:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket"
          ],
          "Resource": "arn:aws:s3:::<s3_bucket_name>"
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:AbortMultipartUpload"
          ],
          "Resource": "arn:aws:s3:::<s3_bucket_name>/*"
        }
      ]
    }
    ```

    The role needs to list the bucket, write and read staged objects, delete validation and completed objects, and abort incomplete multipart uploads.

    You do not need to give the Databricks service principal an AWS IAM role for this staging path. Confidence supplies short-lived S3 credentials when it runs `COPY INTO`.

    ### Enter the S3 settings in Confidence

    In the Databricks connector form, select **AWS - S3** and enter:

    * **Region**: the AWS region of the bucket.
    * **Role ARN**: the IAM role that Confidence assumes.
    * **Bucket**: the S3 bucket name, without an `s3://` prefix.
    * **Authenticate with credentials**: leave unchecked for web identity authentication; select it only when using an IAM access key and secret.
  </Tab>

  <Tab title="Azure ADLS Gen2">
    Confidence writes Parquet files to an ADLS Gen2 filesystem and creates a short-lived user delegation SAS for Databricks to read during `COPY INTO`. Azure workload identity federation lets Confidence exchange a Google service account token for an Azure token without storing an Azure client secret.

    ### Create or reuse the storage account and filesystem

    1. Create or reuse an Azure storage account with hierarchical namespace enabled.
    2. Create or reuse a filesystem. Azure may also call a filesystem a container.
    3. Choose an optional path prefix for Confidence staging. The prefix is relative to the filesystem, for example `confidence/staging`. Do not enter an `abfss://` URL or a leading storage-account URL.

    ### Create a user-assigned managed identity

    1. In the Azure portal, create a user-assigned managed identity.
    2. Open **Federated credentials** for the identity and add a credential for Google Cloud.
    3. Configure the credential with these values:

    | Field | Value |
    | - | - |
    | Issuer | `https://accounts.google.com` |
    | Subject | The **Your service account ID** shown in the Confidence ADLS form |
    | Audience | `api://AzureADTokenExchange` |

    The subject is the unique ID of the Confidence service account, not the service account email address. For more information, see Microsoft's guide to [workload identity federation with Google Cloud](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-google-cloud).

    The managed identity's client ID is the value you enter in Confidence. The Microsoft Entra tenant ID is the directory tenant that contains the managed identity.

    ### Grant storage permissions

    Assign the managed identity **Storage Blob Data Contributor** on the storage account or filesystem used for staging. This role allows Confidence to list, read, write, and delete the staged objects.

    Confidence also obtains a user delegation key to create the short-lived SAS that Databricks uses. If the data role is assigned only at filesystem scope, assign **Storage Blob Delegator** at the storage account, resource group, or subscription scope so the identity can request that key. A data role assigned at storage-account scope may already include the required action.

    If your storage account relies on Data Lake Storage ACLs in addition to Azure RBAC, grant the identity execute permission on each parent directory and read/write permission on the staging path. See Microsoft's guidance for [authorizing access to Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory) and [ADLS Gen2 access control](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control).

    ### Enter the ADLS settings in Confidence

    In the Databricks connector form, select **Azure - ADLS Gen2** and enter:

    * **Storage account**: the storage account name only, without a URL.
    * **Filesystem**: the filesystem or container name.
    * **Path prefix**: an optional relative path, or leave it empty to use the filesystem root.
    * **Microsoft Entra tenant ID**: the tenant that contains the managed identity.
    * **Managed identity/application client ID**: the client ID of the user-assigned managed identity with the federated credential.

    Do not enter an Azure client secret for staging. The Azure identity is federated with the Confidence service account. The Databricks service principal still needs its OAuth client ID and secret in the shared Databricks connection settings.

    You do not need to configure a Databricks storage credential or external location for this staging path. Confidence supplies a short-lived SAS token when it runs `COPY INTO`.

    For more information, see Microsoft's documentation for [workload identity federation](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation) and [user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas).

    If Azure validation fails, check the federated identity credential first. The issuer, subject, and audience must match the Google token, and the subject must be the service account ID shown in the form. Then verify the managed identity's storage role assignments and the Databricks service principal's warehouse and Unity Catalog permissions.
  </Tab>
</Tabs>

## Step 6: Configure the Metrics Data Warehouse

The Metrics Data Warehouse uses the shared Databricks service principal and SQL warehouse. It does not use staging storage.

1. In the Confidence app, select **Admin > Connections > Metrics Data Warehouse**.
2. Select **Databricks**.
3. Enter the Databricks **Server hostname** in **Host (excluding protocol)**. Then enter the **Warehouse ID**, **Client ID**, and **Client Secret** from the earlier steps.
4. Enter the **Exposure schema** and **Results schema**. Use fully qualified Unity Catalog names such as `main.confidence_exposure` and `main.confidence_results`.
5. Click **Save**.

The service principal needs `USE CATALOG`, `USE SCHEMA`, and `SELECT` on the tables that Confidence reads for metrics. It also needs the appropriate write permissions on the results schema. You can use the same schema for both fields, or keep exposure and results data in separate schemas.

## Step 7: Configure an Event or Flag Applied connector

For an Event or Flag Applied connector, Confidence writes Parquet files to the staging provider and then loads them into Databricks tables. The connector form requires exactly one staging provider.

### Event connector

1. In the Confidence app, select **Admin > Connections > Events**.
2. Create an event connection and select **Databricks** as the destination.
3. Enter the fully qualified **Schema**, shared Databricks connection settings, staging provider settings, and batch settings.
4. Select **Validate**. Confidence checks storage access, Databricks access, table creation, and a test load before allowing you to continue.
5. Resolve any validation errors and save the connection.

Events are written to tables in the configured schema, typically one table per event definition.

### Flag Applied connector

1. In the Confidence app, select **Admin > Connections > Flag Applied**.
2. Create a connection and select **Databricks** as the destination.
3. Enter the assignment **Table**, fully qualified **Schema**, shared Databricks connection settings, staging provider settings, and batch settings.
4. Select **Validate**. Confidence checks storage access, Databricks access, table creation, and a test load before allowing you to continue.
5. Resolve any validation errors and save the connection.

The Databricks service principal needs permission to create the assignment table in the configured schema. If the table already exists and is owned by another principal, grant the service principal `MODIFY` on that table.

## Step 8: Configure an assignment table

To use Flag Applied data in exposure calculations, create an assignment table that points to the Databricks table.

<Steps>
  <Step title="Open the Flag Applied connection">
    In the Confidence app, select **Admin > Connections > Flag Applied** and open the Databricks connection you created.
  </Step>

  <Step title="Create an assignment table">
    In the **Assignment tables** section, click **Create**.
  </Step>

  <Step title="Create or select an entity">
    Create a new entity or select an existing entity. Entities are the things you experiment on, such as users. Enter an entity name such as `User` and select the type of the identifier that identifies the entity.
  </Step>

  <Step title="Enter the assignment table name">
    Enter the Databricks table name, such as `flag_applied`.
  </Step>

  <Step title="Create the assignment table configuration">
    Click **Create**.
  </Step>
</Steps>

## Troubleshooting validation

Validation runs a small end-to-end check. It verifies that Confidence can access the staging location, write a temporary object, connect to the Databricks workspace, use the selected warehouse, inspect the schema, create a table, and load test data. It removes the temporary validation data afterward.

If validation fails, start by checking the selected staging provider's identity and storage permissions. Then verify the Databricks service principal's warehouse and Unity Catalog permissions.

## Related resources

<CardGroup cols={2}>
  <Card title="Metrics introduction" href="/docs/capabilities/metrics/overview">
    Overview of metrics in Confidence
  </Card>

  <Card title="Fact tables" href="/docs/capabilities/metrics/fact-tables">
    Configure fact tables for metrics
  </Card>

  <Card title="Assignment tables" href="/docs/capabilities/metrics/assignment-tables">
    Set up assignment tracking
  </Card>

  <Card title="Configure metric quickstart" href="/docs/guides/create-your-first-metric">
    Create your first metric
  </Card>
</CardGroup>