Skip to main content

Performance Optimization

Slim Diff​

By default, Datafold diffs all modified models and downstream models. However, it won't make sense for all organizations to diff every downstream table every time you make a code update. Tradeoffs of time, cost, and risk must be considered.

That's why we created Slim Diff.

With Slim Diff enabled, Datafold will only diff models with code changes in your Pull Request (PR).

Setting up Slim Diff​

You can turn on Slim Diff in the Datafold app by navigating to Settings → Integrations → Orchestration → [Select your Orchestration] and check the Slim Diff box.


Screen Shot 2022-12-20 at 9 50 03 PM

Diffing only modified models​

With this setting turned on, only the modified models will be diffed by default.


Screen Shot 2022-12-20 at 9 58 55 PM

Diff individual downstream models​

Once Datafold has diffed only the modified models, you still have the option of diffing individual downstream models right within your PR.


Screen Shot 2022-12-20 at 10 00 17 PM

Diff all downstream models​

You can also add the datafold:diff-all-downstream label within your PR, which will automatically diff all downstream models.


Screen Shot 2022-12-20 at 10 03 23 PM

Explicitly define which models to always diff​

Finally, with Slim Diff turned on, there might be certain models or subdirectories that you want to always diff when downstream. You can think of this as an exclusion to the Slim Diff behavior.

Apply the slim_diff: diff_when_downstream meta tag to individual models or entire folders in your dbt_project.yml file:

models:
  <project_name>:
    <directory_name>:   
      +materialized: view
      <model_name>:
        +meta:
          datafold:
            datadiff:
              slim_diff: diff_when_downstream
    
    <directory_name>:
      +meta:
        datafold:
          datadiff:
            slim_diff: diff_when_downstream

These meta tags can also be added in individual yaml files or in config blocks. More details about using meta tags are available in the dbt docs.

With this configuration in place, Slim Diff will prevent downstream models from being run unless they have been designated as exceptions with the slim_diff: diff_when_downstream dbt meta tag.


Screen Shot 2022-12-20 at 10 09 50 PM

As usual, once the PR has been opened, you'll still have the option of diffing individual downstream models that weren't diffed, or diffing all downstream models using the datafold:diff-all-downstream label.