Skip to main content

AWS Aurora Postgres Column-level Lineage

This will guide you through setting up Column-level Lineage with AWS Aurora using CloudWatch. AWS Aurora Serverless is not tied to any (virtual) machine, and therefore it only allows you to send the logs to CloudWatch.

Steps to complete:

  1. Setup Postgres with Permissions
  2. Increase the logging verbosity of Postgres so Datafold can parse lineage
  3. Set up an account for fetching the logs from CloudWatch.
  4. Configure your data source in Datafold

Run SQL Script

To connect to Postgres, you need to create a user with read-only access to all tables in all schemas, write access to Datafold-specific schema for temporary tables:

/* Datafold utilizes a temporary dataset to materialize scratch work and keep data processing in the your warehouse. */

CREATE SCHEMA datafold_tmp;

/* Create a datafold user */


/* Give the datafole role write access to the temporary schema */

GRANT ALL ON SCHEMA datafold_tmp TO datafold;

/* Make sure that the postgres user has read permissions on the tables */

GRANT USAGE ON SCHEMA <myschema> TO datafold;

Increase logging verbosity

To begin, navigate to your database instances in the Amazon RDS sidebar.

Then, create a new Parameter Group. Database instances run with default parameters that do not include logging verbosity. To turn on the logging verbosity, you'll need to create a new Parameter Group. Hit Parameter Groups on the menu and create a new Parameter Group.

Next, select the aurora-postgresql10 parameter group family. This depends on the cluster that you're running. For Aurora serverless, this is the appropriate family.

Finally, set the log_statement enum field to mod - meaning that it will log all the DDL statements, plus data-modifying statements. Note: This field isn't set by default.

After saving the parameter group, go back to your database, and select the database cluster parameter group.

Connect Datafold to CloudWatch

Start by creating a new user to isolate the permissions as much as possible. Go to IAM and create a new user.

Next, create a new group named CloudWatchLogsReadOnly and attach the CloudWatchLogsReadOnlyAccess policy to it. Next, select the group.

When reviewing the user, it should have the freshly created group attached to it.

After confirming the new user you should be given the Access Key and Secret Key. Save these two codes securely to finish configurations on Datafold.

The last piece of information Datafold needs is the CloudWatch Log Group. You will find this in CloudWatch under the Log Group section in the sidebar. It will be formatted as /aws/rds/cluster/<my_cluster_name>/postgresql.

Configure in Datafold

Field NameDescription
NameA name given to the data source within Datafold
HostThe hostname address for your database; default value
PortPostgres connection port; default value is 5432
UserThe user role created in the SQL script; datafold
PasswordThe password created in the SQL permissions script
Database NameThe name of the Postgres database you want to connect to
AWS Access KeyThe Access Key provided in the Connect Datafold to CloudWatch step
AWS SecretThe Secret Key provided in the Connect Datafold to CloudWatch step
Cloudwatch Postgres Log GroupThe path of the Log Group; formatted as /aws/rds/cluster/<my_cluster_name>/postgresql
Schema for temporary tablesThe schema created in the SQL setup script; datafold_tmp

Click Create. Your data source is ready!