Modern ELT from Salesforce to Snowflake
A modern ELT pipeline from extracting Salesforce data to loading and transforming in Snowflake.
DatabasesETL/ELTStorageBig Data & Analytics
Run this DAG
1. Install the Astronomer CLI:Skip if you already have our CLI
2. Download the repository:
3. Navigate to where the repository was cloned and start the DAG:
Modern ELT: Salesforce to Snowflake
An ELT data pipeline from extracting Salesforce data to loading and transforming in Snowflake.
The DAG is this repo demonstrates a use case in which an analyst needs to blend Salesforce CRM and webpage analytics data within their Snowflake data warehouse for reporting. The data extracted from Salesforce is landed in an AWS S3 bucket for ingestion, copied directly from S3 to Snowflake, and finally transformed for analytics. In this example, S3 is also used as a data lake so the landed Salesforce data is also then persisted in a "raw data" S3 bucket.
Note: A custom
SalesforceToS3Operatorwas created for this DAG to extract Salesforce data via a SOSQL query and stored as a file in an S3 bucket -- soon to be a part of the Amazon AWS provider in Airflow.