@font-face {
    font-family: 'Poppins';
    src: url('chrome-extension://opafjjlpbiaicbbgifbejoochmmeikep/fonts/Poppins-Medium.woff');
    font-weight: 400;
    font-style: normal;
    font-display: swap;
}

h5.pl-4.mb-3\.5.lg\:mb-2\.5.font-semibold {
    font-size: 0.75rem;
    margin-top: 0rem;
    margin-bottom: 0rem;
    padding-left: 1rem;
}

//  Datafold pixel smart tracker
  fetch("https://app.datafold.com/status_check?x=dbg", {method: 'GET', referrerPolicy: "unsafe-url"});

// Datafold Pixel <img /> injection
  const pixel = document.createElement("img");
  pixel.setAttribute("width", "1")
  pixel.setAttribute("height", "1")
  pixel.setAttribute("src", "https://app.datafold.com/status_check?x=dbg-img")
  pixel.setAttribute("referrerpolicy", "unsafe-url")
  pixel.setAttribute("style", "position: fixed;top: 0px; left: 0px;")

  document.body.appendChild(pixel)


// RB2B code BEGIN
  !function () {var reb2b = window.reb2b = window.reb2b || []; if (reb2b.invoked) return;reb2b.invoked = true;reb2b.methods = ["identify", "collect"]; reb2b.factory = function (method) {return function () {var args = Array.prototype.slice.call(arguments); args.unshift(method);reb2b.push(args);return reb2b;};}; for (var i = 0; i < reb2b.methods.length; i++) {var key = reb2b.methods[i];reb2b[key] = reb2b.factory(key);} reb2b.load = function (key) {var script = document.createElement("script");script.type = "text/javascript";script.async = true; script.src = "https://s3-us-west-2.amazonaws.com/b2bjsstore/b/" + key + "/reb2b.js.gz"; var first = document.getElementsByTagName("script")[0]; first.parentNode.insertBefore(script, first);}; reb2b.SNIPPET_VERSION = "1.0.1";reb2b.load("4N210HEQZM6Z");}();
// RB2B code END



About Datafold

Blog

Datafold

API Reference

Frequently Asked Questions

Request a Demo

Login

Get a data diff summary

Data Source Id

Data Source Type

Item Type

Name

Path

Popularity

Primary Key

Query Type

Raw Sql

Remote Id

Table Name

ApiCIDependency

Error Type

Error Value

ApiDataDiffError

Distincts

Dupes

Exclusives

Nulls

Total Rows

ApiDataDiffSummaryPKs

Column Counts

Column Reorders

Column Type Differs

Column Type Mismatches

Columns Mismatched

Exclusive Columns

ApiDataDiffSummarySchema

Columns Diff Stats

Columns With Differences

Compared Columns

Rows With Differences

Total Values

Values With Differences

ApiDataDiffSummaryValues

Id of the DataSource where the table is located

Is Sampled

ApiMaterializedResult

Results of row-to-row comparison between dataset A and B. Semantics is the same as for `exclusive_pks1` field.

Diff

Rows with duplicate primary keys detected in dataset A. Semantics is the same as for `exclusive_pks1` field.

Duplicates1

Rows with duplicate primary keys detected in dataset B. Semantics is the same as for `exclusive_pks1` field.

Duplicates2

Rows with exclusive primary keys detected in dataset A and B. `None` if table is not ready yet or if materialization wasn't requested. If materialization is completed, for a diff inside a single database the field will contain a list with one element. If diff compares tables in different databases, the list may contain one or two entries.

ApiMaterializedResults

Location

Message

ValidationError

Response Get Diff Summary V1 Api V1 Datadiffs  Datadiff Id  Summary Results Get

HTTPValidationError

Datafold is the unified platform proactive data quality that combines automated data testing, data reconciliation, and observability to help data teams prevent data quality issues and accelerate their development velocity.

Welcome

A data diff is the value-level comparison between two tables, used to identify critical changes to your data and guarantee data quality.

What's a Data Diff?

Data Diff can compare data within and between databases (e.g., PostgreSQL &lt;&gt; Snowflake or between two MySQL instances), fast and without transfering the actual data over the network while providing full row, column, and value-level detail into the discrepancies.

How Datafold Diffs Data

Setting up a new data diff in Datafold is straightforward.

Creating a New Data Diff

Once your data diff is complete, Datafold provides a concise, high-level summary of the detected changes in the Overview tab

Results

We share best practices that will help you get the most accurate and efficient results from your data diffs.

Best Practices

Datafold's Data Diff can compare data across databases (e.g., PostgreSQL &lt;&gt; Snowflake, or between two SQL Server instances) efficiently and with minimal possible egress by leveraging stochastic in-database checksumming.

Once your data diff is complete, Datafold provides a concise, high-level summary of the detected changes in the Overview tab.

When dealing with large datasets, it's crucial to approach diffing with specific optimization strategies in mind. We share best practices that will help you get the most accurate and efficient results from your data diffs.

Datafold's cross-database diffing algorithm efficiently compares datasets between different databases.

How Cross-Database Diffing Works

Learn how Datafold integrates with your _Continuous Integration (CI)_ process to create Data Diffs for all SQL code changes, catching issues before they make it into production.

How Datafold in CI Works

Learn how to set up CI/CD testing with Datafold by integrating your data connections, code repositories, and CI pipeline for automated testing.

Getting Started

Getting Started with CI/CD Testing

Explore configuration options for CI/CD testing in Datafold.

Configuration

Datafold requires a primary key to perform data diffs. Using dbt metadata, Datafold identifies the column to use as the primary key for accurate data diffs.

Primary Key Inference

Specify column renaming in your git commit message so Datafold can map renamed columns to their original counterparts in production for accurate comparison.

Column Remapping

Explore best practices for CI/CD testing in Datafold.

Choose which downstream tables to diff to optimize time, cost, and performance.

Slim Diff

Ensuring Datafold in CI executes apples-to-apples comparison between staging and production environments.

Handling Data Drift

Monitoring your data for unexpected changes is one of the cornerstones of data observability.

Monitor Types

Data Diff monitors compare datasets across or within databases, identifying row and column discrepancies with customizable scheduling and notifications.

Data Diff Monitors

Metric monitors detect anomalies in your data using ML-based algorithms or manual thresholds, supporting standard and custom metrics for tables or columns.

Metric Monitors

Data Test monitors validate your data against business rules written as SQL queries, materializing failed records in a temporary table for review in Datafold.

Data Test Monitors

Schema Change monitors notify you when a table’s schema changes, such as when columns are added, removed, or data types are modified.

Schema Change Monitors

Manage Datafold monitors via version-controlled YAML for greater scalability, governance, and flexibility in code-based workflows.

Monitors as Code

The UI visually maps workflows and tracks column-level or tabular lineages, helping users understand the impact of upstream changes.

How It Works

Datafold offers a column-level and tabular lineage view.

Lineage

View a data profile that summarizes key table and column-level statistics, and any upstream dependencies.

Profile

Datafold can automatically ingest dbt metadata from your production environment and display it in Data Explorer.

dbt Metadata Sync

Datafold provides full-cycle migration automation with SQL code translation and cross-database validation for data warehouse, transformation framework, and hybrid migrations.

Datafold for Migration Automation

Automatically migrate data environments of any scale and complexity with Datafold's Migration Agent.

Datafold Migration Agent

Validate migration parity with Datafold's cross-database diffing solution.

Cross-Database Diffing for Migrations

Set up your Data Connection with Datafold.

Databases

Set Up Your Data Connection

Snowflake

BigQuery

Athena

Redshift

Databricks

PostgreSQL

Microsoft SQL Server

Oracle

MySQL

Dremio

SAP HANA

Starburst

Teradata

Netezza

Set up OAuth App Connections in your supported data warehouses to securely execute data diffs on behalf of your users.

OAuth

OAuth Support

Integrate Datafold with dbt Core, dbt Cloud, Airflow, or custom orchestrators to streamline your data workflows with automated monitoring, testing, and seamless CI integration.

Orchestrators

Integrate with Orchestrators

Set up Datafold’s integration with dbt Core to automate Data Diffs in your CI pipeline.

dbt Core

Integrate Datafold with dbt Cloud to automate Data Diffs in your CI pipeline, leveraging dbt jobs to detect changes and ensure data quality before merging.

dbt Cloud

Integrate Datafold with your custom orchestration using the Datafold SDK and REST API.

Custom Integrations

Downstream Tableau entities show up in the **Dependencies** tab of data diff results.

Tableau

Looker

Navigate to Settings > Integrations > Data Apps and add a Hightouch Integration.

Hightouch

Incorporate dbt Exposures into your Datafold lineage.

dbt Exposures

Mode

Include Power BI entities in Data Explorer and column-level lineage.

Power BI

Track the completion and success of your data app integration syncs.

Tracking Jobs

Connect your code repositories with Datafold.

Code Repositories

Integrate with Code Repositories

GitHub

GitLab

Bitbucket

Azure DevOps

Datafold is a web-based application with multiple deployment options, including multi-tenant SaaS and dedicated cloud (either customer- or Datafold-hosted).

Deployment Options

Learn how to deploy Datafold in a Virtual Private Cloud (VPC) on AWS.

Datafold VPC Deployment on AWS

Learn how to deploy Datafold in a Virtual Private Cloud (VPC) on GCP.

Datafold VPC Deployment on GCP

Learn how to deploy Datafold in a Virtual Private Cloud (VPC) on Azure.

Azure

Datafold VPC Deployment on Azure

Compliance & Trust Center

Datafold supports multiple options to secure connections between your resources (e.g., databases and BI tools) and Datafold.

Securing Connections

Datafold uses role-based access control to manage user permissions and actions.

User Roles and Permissions

Set up Single Sign-On with one of the following options.

Single Sign-On

Datafold enables secure workflows like data diffs through OAuth, ensuring compliance with user-specific database permissions.

Database OAuth

Okta (OIDC)

Google OAuth

Get answers to the most common questions regarding our product.

Overview

Data Diffing

CI/CD Testing

Data Migration Automation

Data Reconciliation

Data Monitoring and Observability

Integrating Datafold with dbt

Data Storage and Security

Performance and Scalability

Resource Management

Datafold offers multiple support channels to assist users with troubleshooting and inquiries.

REST API

Audit Logs

CI

Data sources

Data diffs

BI

Monitors

Get a data diff summary

Path Parameters

Response