> ## Documentation Index
> Fetch the complete documentation index at: https://docs.datafold.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Connect Datafold to Databricks for data diffing, CI/CD testing, lineage, and migration validation. Includes setup instructions and required permissions.

**NOTE**: Datafold needs catalog-level permissions in your Databricks workspace to read and write table data, stage cross-database diff transfers, query system tables, and deploy migration bundles. You will need workspace admin access to create a service principal and grant the required permissions.

> [!NOTE]
>
> If your workspace storage account is firewalled to block public access, see [Securing Connections → Azure](https://docs.datafold.com/classic/security/securing-connections.md#azure-private-link). Query results over 1 MB are downloaded directly from workspace storage via Cloud Fetch, which typically requires its own private endpoint in addition to the SQL warehouse connection.

**Steps to complete:**

1. [Create a service principal and configure authentication](https://docs.datafold.com/classic/integrations/databases/databricks.md#create-a-service-principal-and-configure-authentication)
2. [Retrieve SQL warehouse connection details](https://docs.datafold.com/classic/integrations/databases/databricks.md#retrieve-sql-warehouse-connection-details)
3. [Grant permissions](https://docs.datafold.com/classic/integrations/databases/databricks.md#grant-permissions)
4. [Configure your data connection in Datafold](https://docs.datafold.com/classic/integrations/databases/databricks.md#configure-in-datafold)

If you plan to run Datafold Migration Agent inference on your own Databricks model serving endpoints, there are two extra prerequisites — see [Databricks AI for migrations](https://docs.datafold.com/classic/integrations/databases/databricks.md#databricks-ai-for-migrations).

## Create a service principal and configure authentication

Create a dedicated service principal for the Datafold integration. This is the identity Datafold will use to connect to your workspace.

1. Go to **Settings** → **Identity and access** → **Service principals**
2. Click **Add service principal** and give it a name (e.g., `datafold`)
3. Select the service principal, go to the **Secrets** tab, and click **Generate secret**
4. Save the **Client ID** and **Secret** — the secret is only shown once

> [!NOTE]
>
> OAuth secrets are valid for up to 730 days. You can have a maximum of 5 active secrets per service principal. Rotate secrets before expiry to avoid connection interruptions.

> [!NOTE]
>
> Datafold also supports Personal Access Tokens as an alternative authentication method. PATs are considered legacy by Databricks — see the [Databricks authentication documentation](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html) for details.

## Retrieve SQL warehouse connection details

Navigate to **SQL Warehouses** under the **SQL** section in the left sidebar.

Choose the preferred warehouse and copy the following fields from its **Connection Details** tab:

* **Server hostname**
* **HTTP path**

You also need to grant the service principal access to the SQL warehouse:

1. On the warehouse page, click the **Permissions** tab
2. Add the service principal and grant **Can Use** permission

## Grant permissions

Run the following SQL statements to grant Datafold the permissions it needs. Replace `<catalog_name>` and `<service_principal_id>` with your values.

> [!NOTE]
>
> The `<service_principal_id>` is the application ID (also called Client ID) of your service principal. In Databricks SQL, service principal identifiers must be enclosed in backticks.

```sql theme={null}
-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT CREATE TABLE ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT MODIFY ON CATALOG <catalog_name> TO `<service_principal_id>`;

-- Temporary schema for Datafold
CREATE SCHEMA IF NOT EXISTS <catalog_name>.datafold_tmp;
GRANT CREATE VOLUME ON SCHEMA <catalog_name>.datafold_tmp TO `<service_principal_id>`;

-- System tables access
GRANT USE CATALOG ON CATALOG system TO `<service_principal_id>`;
GRANT USE SCHEMA ON CATALOG system TO `<service_principal_id>`;
GRANT SELECT ON CATALOG system TO `<service_principal_id>`;
```

### (Optional) Additional grant for migrations

If Datafold delivers a migration on this connection, also grant permission to create schemas: Datafold deploys migrated code into new schemas. This is not needed for data diffing, monitoring, or lineage.

```sql theme={null}
GRANT CREATE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;
```

> [!NOTE]
>
> Authentication for migrations depends on the method. A service principal using **M2M OAuth** needs no additional token scope. If you use a **Personal Access Token** (legacy) instead, it must have the `all-apis` scope, because bundle deployment uploads files via the Databricks workspace-files API, which Databricks gates on `all-apis` even when narrower scopes are present.

## Databricks AI for migrations

> [!NOTE]
>
> This section applies only if the Datafold Migration Agent will run inference on **your** Databricks model serving endpoints. Skip it for data diffing, monitoring, lineage, and CI, and for migrations that use Datafold-managed inference or a different [LLM provider](https://docs.datafold.com/classic/integrations/llm-providers.md).

Databricks-served inference reuses the data connection above — there is no second service principal, endpoint URL, or API key to create. Two things still need arranging on the Databricks side.

### Grant Can Query on the serving endpoints

Serving endpoint access is a separate ACL from SQL warehouse and Unity Catalog access, so the grants above do not cover it. For each endpoint you approve for inference:

1. Navigate to **Serving** under the **Machine Learning** section in the left sidebar
2. Open the endpoint and click its **Permissions** tab
3. Add the Datafold service principal and grant **Can Query**

> [!NOTE]
>
> Inference requires the connection to authenticate as a service principal — **M2M OAuth**, a **Personal Access Token**, or **Azure Entra ID**. Per-user OAuth connections cannot be used, because the agent runs without a user context.

If your workspace serves pay-per-token foundation models as Unity Catalog model services (`system.ai.*`), Datafold lists them from the Unity Catalog model services API. When the service principal cannot read that API, those models are missing from the model picker; endpoints already configured keep serving inference.

### Raise the AI Gateway throughput limit

The default AI Gateway limit is roughly 200,000 input tokens per minute, which throttles a real migration heavily. Ask your Databricks account team to raise it for the models you approve — request at least **2–3M input tokens per minute**. Tier 2 allows up to 10M.

> [!NOTE]
>
> Raising the limit is free: the default is a guardrail, not a paid tier. Databricks typically takes a few days to apply the change, so request it early.

Once **Can Query** is granted and the limit is raised, connect the endpoints in Datafold by adding the [Databricks LLM provider integration](https://docs.datafold.com/classic/integrations/llm-providers/databricks.md).

## Configure in Datafold

Select **M2M OAuth / Service Principal (Recommended)** as the authentication method and fill in the following fields:

| Field | Description |
| - | - |
| Connection name | A name for this data connection within Datafold |
| Host | The Server hostname from the warehouse Connection Details tab |
| HTTP path | The HTTP path from the warehouse Connection Details tab |
| Authentication method | Select **M2M OAuth / Service Principal (Recommended)** |
| Client ID | The Client ID of the service principal |
| Client Secret | The secret generated in the authentication step |
| Catalog | The default catalog name (e.g., `hive_metastore` or your Unity Catalog name) |
| Schema path for temporary tables | The temp schema as `<catalog_name>.datafold_tmp` (e.g., `demo.datafold_tmp`) |

Alternatively, select **Azure Entra ID (Service Principal)** as the authentication method and provide **Azure Tenant ID**, **Azure Client ID**, **Azure Client Secret**, and **Azure Scope** (default `2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default`).

Click **Create**. Your data connection is ready!

## Validate your setup

Run these queries to verify that permissions are configured correctly:

```sql theme={null}
-- Check all grants for the service principal
SHOW GRANTS TO `<service_principal_id>`;
```

```sql theme={null}
-- Verify warehouse access (run from the SQL warehouse the service principal will use)
SELECT 1;
```

```sql theme={null}
-- Verify temp schema access
SHOW TABLES IN <catalog_name>.datafold_tmp;
```

```sql theme={null}
-- Verify system table access (should return a row)
SELECT * FROM system.query.history LIMIT 1;
```

```sql theme={null}
-- Verify volume creation in the temp schema (run as the service principal)
CREATE VOLUME IF NOT EXISTS <catalog_name>.datafold_tmp.datafold_permission_check;
DROP VOLUME IF EXISTS <catalog_name>.datafold_tmp.datafold_permission_check;
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.