Dremio
info
Column-level Lineage is not currently supported for Dremio.
info
Schemas for tables in external data sources need to be specified with quotes e.g., "Postgres prod.analytics.sales".
Steps to complete:
- Configure user in Dremio
- Configure your data source in Datafold
- Create a space for Datafold's temporary views
Configure user in Dremio
To connect to Dremio, create a user with read-only access to all data sources you wish to diff and generate an access token.
Temporary tables will be created in the $scratch schema that doesn't require special permissions.
Configure in Datafold
| Field Name | Description |
|---|---|
| Connection name | A name given to the data source within Datafold. |
| Host | The hostname for your Dremio instance (data.dremio.cloud for Dremio SaaS). |
| Port | Dremio endpoint port; default value is 433. |
| Encryption | Should be checked for Dremio Cloud, possibly unchecked for local deployments. |
| User ID | User ID as created in Dremio, typically an email address. |
| Project ID | Dremio Project UID. If left blank, the default project will be used. |
| Token | Access token generated in Dremio. |
| Password | Alternatively, provide a password. |
| Schema for temporary views | A Dremio space for temporary views. |
| Schema for temporary tables | $scratch should suit most applications, or use "<Datasource>.<schema>" (with quotes) if you wish to create temporary tables in an external datasource. |
Click Create. Your data source is now ready!