Astro Billing ETL
Example DAG showcasing a billing data processing ETL use case implemented with astro library functions.
Run this DAG
1. Install the Astronomer CLI:Skip if you already have our CLI
2. Download the repository:
3. Navigate to where the repository was cloned and start the DAG:
This repo contains several example DAGs highlighting the use of decorators, the TaskFlow API, and the
astro library in Airflow 2. There are also several "classic" DAGs to highlight "before" examples of what these workflows looked like prior to the features being highlighted.
taskflow.py: Shows a basic ETL example using the TaskFlow API and Python decorator to easily pass data between tasks.
classic-python-operator.pyimplements the same DAG as it would have been prior to Airflow 2 and the TaskFlow API.
astro-billing-etl.py: Shows an ETL example for processing billing data by implementing
astrolibrary functions for SQL and Pandas transformations.
classic-billing-etl.pyimplements the same DAG without the use of the
astro-animal-adoptions-etl.py: Provides another ETL example using
The easiest way to run these example DAGs is to use the Astronomer CLI to get an Airflow instance up and running locally:
- Install the Astronomer CLI
- Clone this repo somewhere locally and navigate to it in your terminal
- Initialize an Astronomer project by running
astro dev init
- Start Airflow locally by running
astro dev start
- Navigate to localhost:8080 in your browser and you should see the tutorial DAGs there