Skip to main content
Beta ProductSQL Proxy is currently in beta. Features and APIs may change. Contact your Datafold representative for access.

​
What is SQL Proxy?

SQL Proxy is a middleware that routes SQL queries to different compute resources based on query characteristics. It works with any tool that connects via ODBC/JDBC, including BI tools and dbt. Supported platforms:
  • Databricks (available now)
  • Snowflake (coming soon)

​
The Problem

Without intelligent routing, each system typically connects to a dedicated warehouse sized for peak load—wasting compute on small queries or causing spills on large ones. Oversized warehouses waste compute. Undersized warehouses spill to disk, hurting performance.

​
With Datafold SQL Proxy

Queries are routed to appropriately-sized compute. Large warehouses only spin up for truly large queries. Small workloads run on cheaper compute. Key points:
  • User credentials (PAT or M2M OAuth) are passed through to Databricks
  • Datafold uses a separate admin account to manage infrastructure

​
Routing Modes

ModeDescription
Explicit RoutingControl routing via @datafold: SQL comments
Smart RoutingML-based automatic warehouse selection

​
Authentication

Both methods are passthrough - your credentials are forwarded to Databricks for validation.
MethodDescription
PATDatabricks Personal Access Token
M2M OAuthDatabricks service principal credentials
See Authentication for setup details.

​
Networking Requirements

SQL Proxy is deployed in Datafold’s infrastructure. Your data platform must allow inbound connections from Datafold’s IP ranges. Required Databricks Access:
  • SQL Warehouse connectivity (port 443)
  • Jobs API access (for jobs compute routing)
  • Unity Catalog access (if used)

​
Quick Start

  1. Allowlist Datafold IPs in your Databricks networking config
  2. Provide Datafold admin credentials (see Authentication)
  3. Update connection - point dbt/BI tools to SQL Proxy endpoint
  4. Test connectivity - run dbt debug or a simple query
  5. Add annotations - use @datafold: directives for routing control
# dbt profiles.yml
my_project:
  outputs:
    prod:
      type: databricks
      host: sqlproxy.your-company.datafold.com
      http_path: /sql/1.0/warehouses/proxy
      token: "{{ env_var('DATABRICKS_TOKEN') }}"  # Your existing Databricks PAT
See dbt Integration for complete setup instructions.