Skip to main content

What's a data diff?

A data diff is the value-level comparison between two tables—used to identify critical changes to your data and guarantee data quality.

data diff

Datafold's data-diff is a tool that compares datasets fast, within or across databases.

There is a lot you can do with data-diff: you can test SQL code by comparing development or staging environment data to production, or compare source and target data to identify discrepancies when moving data between databases.

Why diff data?

Just as diffing code and text is fundamental to software engineering and working with text documents, diffing data is essential to the data engineering workflow.

In data engineering, both data and the code that processes it are constantly evolving. Without the ability to easily diff data, understanding and tracking data changes becomes challenging. This slows down the developement process and makes it harder to ensure data quality.

What is Datafold Cloud?

We believe that diffing is a fundamental capability in data engineering that every engineer should have access to. Datafold Cloud is the enterprise-ready solution for comparing datasets within or across databases at scale. It includes comprehensive, optimized, and automated diffing solutions, API access, and secure deployment options.

Why choose Datafold Cloud?

Datafold Cloud provides end-to-end solutions for automating testing, including column-level lineage, ML-based anomaly detection, and enterprise-scale infrastructure support. It caters to complex and production-ready scenarios, including:

  • Automated and collaborative diffing and testing for data transformations in CI
  • Data diffing informed by column-level lineage, and validation of code changes with visibility into BI applications
  • Validating large data migrations or continuous replications with automated cross-database diffing capabilities

Here's a high-level overview of what Datafold Cloud offers:

Feature CategoryDatafold Cloud
Database Support
Databases that are supported for source-destination diffing
Any SQL database, inquire about specific support
Size of datasets supported for diffing
Unlimited with advanced performance optimization
Primary Key Data Type Support
Data types of primary keys that are supported for diffing
Numerical, string, datetime, boolean, composite
Data Types Diffing Support
Data types that are supported for per-column diffing
All data types
Export Diff Results to Database
Materialize diffing results in your database of choice
Value-level diffs
Investigate row-by-row column value differences between source and destination databases
✅ (JSON & GUI)
Diff UI
Explore diffs visually and easily share them with your team and stakeholders
API Access
Automatically create diffs and receive results at scale using the Datafold REST API
Persisting Diff History
Persist the result history of diffs to know how your data and diffs have changed over time
Scheduled Checks
Run scheduled diffs for a defined list of tables
Receive automatic alerts about detected discrepancies between tables (Coming Soon)
✅ (Coming soon)
Security and Compliance
Run diffs in secure and compliant environments
HIPAA, SOC2 Type II, GDPR compliant
Deployment Options
Deploy your diffs in secure environments that meet your security standards
Multi-tenant SaaS or Single-tenant in VPC
Choose which channels offer the greatest support to your use cases and users
Enterprise support from Datafold team members
The types of SLAs that exist to guarantee your team can diff and interact with diffs as expected
✅ (Coming soon)