Slim Diff
Choose which downstream tables to diff to optimize time, cost, and performance.
By default, Datafold diffs all modified models and downstream models. However, it won’t make sense for all organizations to diff every downstream table every time you make a code update. Tradeoffs of time, cost, and risk must be considered.
That’s why we created Slim Diff.
With Slim Diff enabled, Datafold will only diff models with dbt code changes in your Pull Request (PR).
Setting up Slim Diff
You can turn on Slim Diff in the Datafold app by navigating to Settings → Integrations → Orchestration → [Select your Orchestration] and check the Slim Diff box.
Diffing only modified models
With this setting turned on, only the modified models will be diffed by default.
Diff individual downstream models
Once Datafold has diffed only the modified models, you still have the option of diffing individual downstream models right within your PR.
Diff all downstream models
You can also add the datafold:diff-all-downstream
label within your PR, which will automatically diff all downstream models.
Explicitly define which models to always diff
Finally, with Slim Diff turned on, there might be certain models or subdirectories that you want to always diff when downstream. You can think of this as an exclusion to the Slim Diff behavior.
Apply the slim_diff: diff_when_downstream
meta tag to individual models or entire folders in your dbt_project.yml
file:
models:
<project_name>:
<directory_name>:
+materialized: view
<model_name>:
+meta:
datafold:
datadiff:
slim_diff: diff_when_downstream
<directory_name>:
+meta:
datafold:
datadiff:
slim_diff: diff_when_downstream
These meta tags can also be added in individual yaml files or in config blocks. More details about using meta tags are available in the dbt docs.
With this configuration in place, Slim Diff will prevent downstream models from being run unless they have been designated as exceptions with the slim_diff: diff_when_downstream
dbt meta tag.
As usual, once the PR has been opened, you’ll still have the option of diffing individual downstream models that weren’t diffed, or diffing all downstream models using the datafold:diff-all-downstream
label.