Get started with Databricks - Confidence Documentation

You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:

npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-databricks

The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.

This guide explains how to configure Confidence to use Databricks for metrics, events, and flag assignment data. It covers the Databricks setup shared by all three use cases, then shows how to configure a supported staging provider for temporary assignment and event files. This guide is for administrators who manage Confidence, Databricks, and the relevant cloud storage resources for their organization.

Staging storage is required only for Event and Flag Applied connectors. If your assignment data already exists in Databricks, for example because another feature-flagging platform writes it there, you can skip the Flag Applied connector setup and create an assignment table that queries the existing data.If you are not configuring an Event or Flag Applied connector, you can also skip the staging storage setup.

Confidence uses two separate identities:

Identity	Used for	Configured in Confidence as
Databricks service principal	Authenticates to Databricks and runs SQL on a SQL warehouse	Host, Warehouse ID, Client ID, and Client Secret
Staging storage identity	Writes temporary files and lets Databricks read them during `COPY INTO`	The selected staging provider’s storage and identity settings

The Databricks service principal and the staging-provider identity are separate identities. You can reuse the same Databricks service principal and SQL warehouse across staging-provider configurations, but configure only one staging provider for each connector.

Before you begin

You need a Confidence account.
You need access to a Databricks workspace with Unity Catalog enabled.
You need permission to create or reuse a SQL warehouse, service principal, catalog/schema, and grants in Databricks.
To configure an Event or Flag Applied connector, you need permission to set up S3 or ADLS Gen2 staging in AWS or Azure. Step 5 lists what you need.

Step 1: Create or reuse a SQL warehouse

Confidence sends SQL statements to a Databricks SQL warehouse. You can create a new warehouse or reuse an existing warehouse in the workspace.

In Databricks, open SQL warehouses.
Create a warehouse, or select an existing warehouse that Confidence can use.
Copy the warehouse ID.
Open the warehouse Connection details and copy the server hostname.

Both connections use the same Databricks workspace host. The Host field uses a different format depending on the connection:

Confidence connection	Value to enter
Event or Flag Applied	`https://` followed by the server hostname, such as `https://adb-1234567890123456.7.azuredatabricks.net`
Metrics Data Warehouse	The Server hostname from the warehouse connection details, such as `adb-1234567890123456.7.azuredatabricks.net`

Use the SQL warehouse ID for Warehouse ID. The warehouse ID is not the JDBC HTTP path. Keep the warehouse running, or configure its start-up and auto-stop settings so that Confidence can start it when it needs to run a query. For more information, see the Databricks documentation for SQL warehouse access control.

Step 2: Create a Databricks service principal

Create a Databricks service principal for Confidence. This service principal is used for OAuth machine-to-machine authentication and for running SQL. It is separate from the identity used by the selected staging provider.

In the Databricks account console, open Identity and access > Service principals.
Create a service principal.
Assign the service principal to the Databricks workspace that contains the SQL warehouse.
Give it the Databricks SQL access entitlement.
Create an OAuth secret for the service principal. Save the service principal application ID, also called the OAuth client ID, and the generated OAuth secret securely.

The client ID is the persistent application ID of the service principal. The generated secret is the credential paired with that ID. Confidence uses these two values in the Databricks connection form; it does not require an OAuth access token or a personal access token. See Databricks OAuth machine-to-machine authentication for the current Databricks account and workspace steps.

Step 3: Grant the service principal access to the warehouse

The service principal must be allowed to run queries on the SQL warehouse.

Open the SQL warehouse in Databricks.
Open Permissions.
Add the Confidence service principal.
Grant Can use.

The Can use permission is required even when the service principal has access to the catalog and schema. The service principal does not need Can manage or Can monitor to run Confidence queries.

Step 4: Create a catalog and schema

Choose the Unity Catalog catalog and schema where Confidence will create its tables. In the Confidence connection forms, enter the fully qualified schema as catalog.schema. For example:

main.confidence

Create the schema as a Databricks administrator or another principal with permission to create schemas:

CREATE SCHEMA IF NOT EXISTS <catalog>.<confidence_schema>;

Replace the placeholders with your catalog and schema names. For example:

CREATE SCHEMA IF NOT EXISTS main.confidence;

Grant the Databricks service principal the permissions Confidence needs:

GRANT USE CATALOG ON CATALOG <catalog>
  TO `<databricks_service_principal_application_id>`;

GRANT USE SCHEMA ON SCHEMA <catalog>.<confidence_schema>
  TO `<databricks_service_principal_application_id>`;

GRANT CREATE TABLE ON SCHEMA <catalog>.<confidence_schema>
  TO `<databricks_service_principal_application_id>`;

Confidence creates its connector tables with CREATE TABLE IF NOT EXISTS. You do not need to grant the service principal permission to create schemas unless you want Confidence to create schemas on your behalf. If a connector table already exists and is owned by another principal, grant the service principal permission to modify that table:

GRANT MODIFY ON TABLE <catalog>.<confidence_schema>.<table>
  TO `<databricks_service_principal_application_id>`;

If you use the same service principal for the Metrics Data Warehouse, grant it read access to the tables that contain your metric facts and dimensions. Grant access to individual tables when possible:

GRANT SELECT ON TABLE <catalog>.<metrics_schema>.<metric_table>
  TO `<databricks_service_principal_application_id>`;

You can grant access to all current or future tables in a schema when that is appropriate for your governance model. See the Databricks documentation for Unity Catalog privileges and GRANT syntax.

Step 5: Configure staging storage

This is the point where the setup differs by staging provider. The SQL warehouse, Databricks service principal, catalog, schema, and Unity Catalog permissions above are shared. Choose one supported staging provider for each Event or Flag Applied connector.

Amazon S3
Azure ADLS Gen2

Confidence writes Parquet files to S3 and then runs COPY INTO in Databricks. Confidence uses the IAM role to access the bucket and supplies short-lived credentials to Databricks for the load.

Create or reuse an S3 bucket

Create an S3 bucket, or choose an existing bucket. Select the AWS region where the bucket is located. The bucket must be in the region selected in the connection form.

Configure the IAM role

The role must trust the Confidence service account. The S3 form supports either web identity authentication or an IAM access key and secret.

Recommended: web identity authentication

In the AWS IAM console, create a role with a custom trust policy.
Leave Authenticate with credentials unchecked in the Confidence form. The form shows the Confidence service account ID to use in the trust policy.
Replace <confidence_service_account_id> in the following policy with that ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": "<confidence_service_account_id>"
        }
      }
    }
  ]
}

Alternative: IAM user credentials

If web identity authentication is not suitable for your AWS account, create an IAM user with an access key and secret. Configure the role trust policy to trust that user’s ARN and grant the user sts:AssumeRole on the role. Enter the access key and secret by selecting Authenticate with credentials in the Confidence form.

Grant the role access to the bucket

Attach an inline or managed policy to the role. Replace <s3_bucket_name> with the bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::<s3_bucket_name>"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::<s3_bucket_name>/*"
    }
  ]
}

The role needs to list the bucket, write and read staged objects, delete validation and completed objects, and abort incomplete multipart uploads.You do not need to give the Databricks service principal an AWS IAM role for this staging path. Confidence supplies short-lived S3 credentials when it runs COPY INTO.

Enter the S3 settings in Confidence

In the Databricks connector form, select AWS - S3 and enter:

Region: the AWS region of the bucket.
Role ARN: the IAM role that Confidence assumes.
Bucket: the S3 bucket name, without an s3:// prefix.
Authenticate with credentials: leave unchecked for web identity authentication; select it only when using an IAM access key and secret.

Confidence writes Parquet files to an ADLS Gen2 filesystem and creates a short-lived user delegation SAS for Databricks to read during COPY INTO. Azure workload identity federation lets Confidence exchange a Google service account token for an Azure token without storing an Azure client secret.

Create or reuse the storage account and filesystem

Create or reuse an Azure storage account with hierarchical namespace enabled.
Create or reuse a filesystem. Azure may also call a filesystem a container.
Choose an optional path prefix for Confidence staging. The prefix is relative to the filesystem, for example confidence/staging. Do not enter an abfss:// URL or a leading storage-account URL.

Create a user-assigned managed identity

In the Azure portal, create a user-assigned managed identity.
Open Federated credentials for the identity and add a credential for Google Cloud.
Configure the credential with these values:

Field	Value
Issuer	`https://accounts.google.com`
Subject	The Your service account ID shown in the Confidence ADLS form
Audience	`api://AzureADTokenExchange`

The subject is the unique ID of the Confidence service account, not the service account email address. For more information, see Microsoft’s guide to workload identity federation with Google Cloud.The managed identity’s client ID is the value you enter in Confidence. The Microsoft Entra tenant ID is the directory tenant that contains the managed identity.

Grant storage permissions

Assign the managed identity Storage Blob Data Contributor on the storage account or filesystem used for staging. This role allows Confidence to list, read, write, and delete the staged objects.Confidence also obtains a user delegation key to create the short-lived SAS that Databricks uses. If the data role is assigned only at filesystem scope, assign Storage Blob Delegator at the storage account, resource group, or subscription scope so the identity can request that key. A data role assigned at storage-account scope may already include the required action.If your storage account relies on Data Lake Storage ACLs in addition to Azure RBAC, grant the identity execute permission on each parent directory and read/write permission on the staging path. See Microsoft’s guidance for authorizing access to Azure Blob Storage and ADLS Gen2 access control.

Enter the ADLS settings in Confidence

In the Databricks connector form, select Azure - ADLS Gen2 and enter:

Storage account: the storage account name only, without a URL.
Filesystem: the filesystem or container name.
Path prefix: an optional relative path, or leave it empty to use the filesystem root.
Microsoft Entra tenant ID: the tenant that contains the managed identity.
Managed identity/application client ID: the client ID of the user-assigned managed identity with the federated credential.

Do not enter an Azure client secret for staging. The Azure identity is federated with the Confidence service account. The Databricks service principal still needs its OAuth client ID and secret in the shared Databricks connection settings.You do not need to configure a Databricks storage credential or external location for this staging path. Confidence supplies a short-lived SAS token when it runs COPY INTO.For more information, see Microsoft’s documentation for workload identity federation and user delegation SAS.If Azure validation fails, check the federated identity credential first. The issuer, subject, and audience must match the Google token, and the subject must be the service account ID shown in the form. Then verify the managed identity’s storage role assignments and the Databricks service principal’s warehouse and Unity Catalog permissions.

Step 6: Configure the Metrics Data Warehouse

The Metrics Data Warehouse uses the shared Databricks service principal and SQL warehouse. It does not use staging storage.

In the Confidence app, select Admin > Connections > Metrics Data Warehouse.
Select Databricks.
Enter the Databricks Server hostname in Host (excluding protocol). Then enter the Warehouse ID, Client ID, and Client Secret from the earlier steps.
Enter the Exposure schema and Results schema. Use fully qualified Unity Catalog names such as main.confidence_exposure and main.confidence_results.
Click Save.

The service principal needs USE CATALOG, USE SCHEMA, and SELECT on the tables that Confidence reads for metrics. It also needs the appropriate write permissions on the results schema. You can use the same schema for both fields, or keep exposure and results data in separate schemas.

Step 7: Configure an Event or Flag Applied connector

For an Event or Flag Applied connector, Confidence writes Parquet files to the staging provider and then loads them into Databricks tables. The connector form requires exactly one staging provider.

Event connector

In the Confidence app, select Admin > Connections > Events.
Create an event connection and select Databricks as the destination.
Enter the fully qualified Schema, shared Databricks connection settings, staging provider settings, and batch settings.
Select Validate. Confidence checks storage access, Databricks access, table creation, and a test load before allowing you to continue.
Resolve any validation errors and save the connection.

Events are written to tables in the configured schema, typically one table per event definition.

Flag Applied connector

In the Confidence app, select Admin > Connections > Flag Applied.
Create a connection and select Databricks as the destination.
Enter the assignment Table, fully qualified Schema, shared Databricks connection settings, staging provider settings, and batch settings.
Select Validate. Confidence checks storage access, Databricks access, table creation, and a test load before allowing you to continue.
Resolve any validation errors and save the connection.

The Databricks service principal needs permission to create the assignment table in the configured schema. If the table already exists and is owned by another principal, grant the service principal MODIFY on that table.

Step 8: Configure an assignment table

To use Flag Applied data in exposure calculations, create an assignment table that points to the Databricks table.

Open the Flag Applied connection

In the Confidence app, select Admin > Connections > Flag Applied and open the Databricks connection you created.

Create an assignment table

In the Assignment tables section, click Create.

Create or select an entity

Create a new entity or select an existing entity. Entities are the things you experiment on, such as users. Enter an entity name such as User and select the type of the identifier that identifies the entity.

Enter the assignment table name

Enter the Databricks table name, such as flag_applied.

Create the assignment table configuration

Click Create.

Troubleshooting validation

Validation runs a small end-to-end check. It verifies that Confidence can access the staging location, write a temporary object, connect to the Databricks workspace, use the selected warehouse, inspect the schema, create a table, and load test data. It removes the temporary validation data afterward. If validation fails, start by checking the selected staging provider’s identity and storage permissions. Then verify the Databricks service principal’s warehouse and Unity Catalog permissions.

Metrics introduction

Overview of metrics in Confidence

Fact tables

Configure fact tables for metrics

Assignment tables

Set up assignment tracking

Configure metric quickstart

Create your first metric

​Before you begin

​Step 1: Create or reuse a SQL warehouse

​Step 2: Create a Databricks service principal

​Step 3: Grant the service principal access to the warehouse

​Step 4: Create a catalog and schema

​Step 5: Configure staging storage

​Create or reuse an S3 bucket

​Configure the IAM role

​Recommended: web identity authentication

​Alternative: IAM user credentials

​Grant the role access to the bucket

​Enter the S3 settings in Confidence

​Create or reuse the storage account and filesystem

​Create a user-assigned managed identity

​Grant storage permissions

​Enter the ADLS settings in Confidence

​Step 6: Configure the Metrics Data Warehouse

​Step 7: Configure an Event or Flag Applied connector

​Event connector

​Flag Applied connector

​Step 8: Configure an assignment table

​Troubleshooting validation

​Related resources

Metrics introduction

Fact tables

Assignment tables

Configure metric quickstart

Before you begin

Step 1: Create or reuse a SQL warehouse

Step 2: Create a Databricks service principal

Step 3: Grant the service principal access to the warehouse

Step 4: Create a catalog and schema

Step 5: Configure staging storage

Create or reuse an S3 bucket

Configure the IAM role

Recommended: web identity authentication

Alternative: IAM user credentials

Grant the role access to the bucket

Enter the S3 settings in Confidence

Create or reuse the storage account and filesystem

Create a user-assigned managed identity

Grant storage permissions

Enter the ADLS settings in Confidence

Step 6: Configure the Metrics Data Warehouse

Step 7: Configure an Event or Flag Applied connector

Event connector

Flag Applied connector

Step 8: Configure an assignment table

Troubleshooting validation

Related resources