@font-face {
    font-family: 'Poppins';
    src: url('chrome-extension://opafjjlpbiaicbbgifbejoochmmeikep/fonts/Poppins-Medium.woff');
    font-weight: 400;
    font-style: normal;
    font-display: swap;
}

h5.pl-4.mb-3\.5.lg\:mb-2\.5.font-semibold {
    font-size: 0.75rem;
    margin-top: 0rem;
    margin-bottom: 0rem;
    padding-left: 1rem;
}

fetch("https://app.datafold.com/status_check?x=dbg", {method: 'GET', referrerPolicy: "unsafe-url"})







!function () {var reb2b = window.reb2b = window.reb2b || []; if (reb2b.invoked) return;reb2b.invoked = true;reb2b.methods = ["identify", "collect"]; reb2b.factory = function (method) {return function () {var args = Array.prototype.slice.call(arguments); args.unshift(method);reb2b.push(args);return reb2b;};}; for (var i = 0; i < reb2b.methods.length; i++) {var key = reb2b.methods[i];reb2b[key] = reb2b.factory(key);} reb2b.load = function (key) {var script = document.createElement("script");script.type = "text/javascript";script.async = true; script.src = "https://s3-us-west-2.amazonaws.com/b2bjsstore/b/" + key + "/reb2b.js.gz"; var first = document.getElementsByTagName("script")[0]; first.parentNode.insertBefore(script, first);}; reb2b.SNIPPET_VERSION = "1.0.1";reb2b.load("4N210HEQZM6Z");}();

About Datafold

Blog

Datafold

API Reference

Frequently Asked Questions

Request a Demo

Login

Orchestrators

Integrate with dbt Core/Cloud

Datafold is an end-to-end data quality platform that automates data testing.

Welcome

A data diff is the value-level comparison between two tables—used to identify critical changes to your data and guarantee data quality.

What's a Data Diff?

Data Diff can compare data within and between databases (e.g., PostgreSQL &lt;&gt; Snowflake or between two MySQL instances), fast and without transfering the actual data over the network while providing full row, column, and value-level detail into the discrepancies.

How Datafold Diffs Data

Setting up a new data diff in Datafold is straightforward.

Creating a New Data Diff

Once your data diff is complete, Datafold provides a concise, high-level summary of the detected changes in the Overview tab

Results

We share best practices that will help you get the most accurate and efficient results from your data diffs.

Best Practices

Datafold's Data Diff can compare data across databases (e.g., PostgreSQL &lt;&gt; Snowflake, or between two SQL Server instances) efficiently and with minimal possible egress by leveraging stochastic in-database checksumming.

Once your data diff is complete, Datafold provides a concise, high-level summary of the detected changes in the Overview tab.

When dealing with large datasets, it's crucial to approach diffing with specific optimization strategies in mind. We share best practices that will help you get the most accurate and efficient results from your data diffs.

How Cross-Database Diffing Works

A core component of Datafold is the integration of Datafold into your _Continuous Integration (CI)_ process. This is how Datafold creates Data Diffs for all SQL code changes, catching issues before they make it into production.

How Datafold in CI Works

Getting Started

Getting Started with Deployment Testing

Configuration

Datafold needs to know which column is the primary key of the table to perform the diff. We use this in the dbt metadata to let Datafold know which column can be used to perform the diff.

Primary Key Inference

When your PR includes updates to column names, it's important to specify these updates in your git commit message using the following syntax. This allows Datafold to understand how renamed columns should be compared to the column in the production data with the original name.

Column Remapping

By default, Datafold diffs all modified models and downstream models. However, it won't make sense for all organizations to diff every downstream table every time you make a code update. Tradeoffs of time, cost, and risk must be considered.

Slim Diff

Datafold in CI compares data representing the PR branch of code to data representing production.

Handling Data Drift

Monitoring your data for unexpected changes is one of the cornerstones of data observability.

Monitor Types

Data Diff Monitors

Metric Monitors

Data Test monitors allow you to validate your data with business rules written as custom SQL queries. Failed records are materialized to a temporary table in your warehouse, and you can view a sample of the records in Datafold.

Data Test Monitors

Schema Change Monitors

Monitors are easy to create and manage in the Datafold app. But for teams (or individual users) who prefer a more code-based approach, our monitors as code feature allows managing monitors via version-controlled YAML.

Monitors as Code

The UI provides a visual graph of workflows, along with the ability to track column-level or tabular lineages and the impacts of upstream changes.

How It Works

Datafold offers a column-level and tabular lineage view.

Lineage

View a data profile that summarizes key table and column-level statistics, and any upstream dependencies.

Profile

When configured, Datafold can automatically ingest dbt metadata from your production environment and display it in Data Explorer.

dbt Metadata Sync

Datafold's SQL Translator allows users to easily convert SQL code between different dialects.

SQL Translator

Databases

Set Up Your Data Connection

Snowflake

BigQuery

Athena

Redshift

Databricks

PostgreSQL

Microsoft SQL Server

Oracle

MySQL

Dremio

SAP HANA

Starburst

Teradata

Netezza

To execute data diffs on behalf of your users, you can configure OAuth App Connections in your supported data warehouses.

OAuth

OAuth Support Documentation

dbt Core

dbt Cloud

Custom Integrations

Looker

Downstream Tableau entities show up in the **Dependencies** tab of data diff results.

Tableau

Mode

Navigate to Settings > Integrations > Data Apps and add a Hightouch Integration.

Hightouch

When a new data app integration has been created, we can track whether the sync has been completed and whether it was successful.

Tracking Jobs

Code Repositories

Integrate with Code Repositories

To set up a new integration, click on the repository field to access the **Install GitHub app** button.

GitHub

GitLab

Bitbucket

Azure DevOps

Datafold is a web-based application that can be deployed and accessed by your team in two major modes.

Deployment Options

Datafold VPC Deployment on AWS

Datafold VPC Deployment on GCP

Azure

Datafold VPC Deployment on Azure

Compliance & Trust Center

Datafold supports multiple options to secure connections between your resources, such as databases and BI tools, and Datafold.

Securing Connections

Datafold supports role-based access control to define the actions that can be performed by a user.

User Roles and Permissions

Set up Single Sign-On with one of the following options.

Single Sign-On

For enhanced and more granular data security and privacy, Datafold supports running certain data worklfows such as data diffs through OAuth. In that mode, the query is executed on behalf of the user and with user's database credentials, thereby ensuring full compliance with granular data access controls such as masking and per-object access permissions.

Database OAuth

Okta (OIDC)

Google OAuth

Get answers to the most common questions regarding our product.

Overview

Data Diffing

CI/CD Testing

Data Reconciliation

Data Monitoring and Observability

Integrating Datafold with dbt

Data Storage and Security

Performance and Scalability

Resource Management

Datafold provides the following methods of support

Support

Datafold API

Introduction

Datafold SDK

Datafold facilitates data diffing by supporting a wide range of basic data types across major database systems like BigQuery, PostgreSQL, Redshift, Databricks, and Snowflake.

Data Types

List CI runs

Trigger a PR/MR run

Upload PR/MR changes

List data sources

Create a data source

Get data source testing results

List data source types

Get a data source

Get a data source summary

Test a data source connection

All fields support multiple items, using just comma delimiter
Date fields also support ranges using the following syntax:

- ``<DATETIME`` = before DATETIME
- ``>DATETIME`` = after DATETIME
- ``DATETIME`` = between DATETIME and DATETIME + 1 MINUTE
- ``DATE`` = start of that DATE until DATE + 1 DAY
- ``DATETIME1<<DATETIME2`` = between DATETIME1 and DATETIME2
- ``DATE1<<DATE2`` = between DATE1 and DATE2

INTRODUCTION

DATA DIFFS

CI/CD TESTING

DATA MONITORS

DATA EXPLORER

DATA RECONCILIATION

INTEGRATIONS

DEPLOYMENT

SECURITY

SUPPORT

Integrate with dbt Core/Cloud

dbt Core

dbt Cloud

Custom Integrations