Airflow Guides
Topics
Introduction to Airflow Decorators
An overview of Airflow decorators and how they can improve the DAG authoring experience.
Deferrable Operators
How to implement deferrable operators to save cost and resources with Airflow.
Debugging DAGs
A beginner's guide to figuring out what's going wrong with your Airflow DAGs
Scheduling and Timetables in Airflow
Everything you need to know about scheduling your Airflow DAGs.
Airflow Data Quality Checks with SQL Operators
Executing queries in Apache Airflow DAGs to ensure data quality.
Using Airflow with SageMaker
Methods for orchestrating SageMaker machine learning pipelines with Airflow.
Executing Notebooks with Airflow
Methods for orchestrating commonly used notebooks with Airflow.
Cross-DAG Dependencies
How to implement dependencies between your Airflow DAGs.
Testing Airflow DAGs
How to apply test-driven development practices to your Airflow DAGs.
Using Task Groups in Airflow
Using Task Groups to build modular workflows in Airflow.
Passing Data Between Airflow Tasks
Methods for sharing metadata and information between tasks in your Apache Airflow DAGs, including XCom.
Deploying Kedro Pipelines to Apache Airflow
How to use the kedro-airflow plugin to change your Kedro pipelines into Apache Airflow DAGs and deploy them to a production environment.
Orchestrating Databricks Jobs with Airflow
Orchestrating Databricks Jobs from your Apache Airflow DAGs.
Executing Azure Data Factory Pipelines with Airflow
Triggering remote jobs in Azure Data Factory from your Apache Airflow DAGs.
Executing Azure Data Explorer Queries with Airflow
Executing Azure Data Explorer queries from your Apache Airflow DAGs.
Orchestrating Azure Container Instances with Airflow
Orchestrating containers with Azure Container Instances from your Apache Airflow DAGs.
Get Started with Apache Airflow 2.0
Test Apache Airflow 2.0 on your local machine with the Astronomer CLI.
Using Airflow to Execute SQL
Executing queries, parameterizing queries, and embedding SQL-driven ETL in Apache Airflow DAGs.
Integrating Airflow and Great Expectations
Using the Great Expectations provider natively in your Airflow DAGs.
Understanding the Airflow Metadata Database
An structural walkthrough of Apache Airflow's metadata database, with a full ERD.
Executing Talend Jobs with Airflow
Triggering remote jobs in Talend from your Apache Airflow DAGs.
Integrating Airflow and Hashicorp Vault
Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.
Importing Custom Hooks & Operators
How to correctly import custom hooks and operators.
Scaling Out Airflow
How to tune your Airflow environment so it scales with your DAGs.
Airflow Executors Explained
A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.
Introduction to Kubernetes
High-level overview of introductory concepts in Kubernetes.
Best Practices Calling AWS Lambda from Airflow
A few tips, guidelines, and best practices for calling Lambda from Airflow
Using Kerberos in Apache Airflow
How to use Kerberos and Kerberized hooks in Airflow
Running scripts using the BashOperator
Learn and troubleshoot how to run shell scripts using the Bash Operator in Airflow
Templating in Airflow
How to leverage the power of Jinja templating when writing your DAGs.
Branching in Airflow
Use Apache Airflow's BranchPythonOperator and ShortCircuitOperator to execute conditional branches in your workflow
Airflow's Components
How all of Apache Airflow's components fit together.
Useful SQL queries for Apache Airflow
A home for SQL queries that we frequently run on our Airflow postgres database.
Managing your Connections in Apache Airflow
An overview of how connections work in the Airflow UI.
DAG Writing Best Practices in Apache Airflow
How to create effective, clean, and functional DAGs.
Intro to Apache Airflow DAGs
What are DAGs and how they are constructed in Apache Airflow?
Dynamically Generating DAGs in Airflow
Using a base DAG template to create multiple DAGs.
Editing Task and DAG Metadata
What are DAGs and how they are constructed in Apache Airflow?
Error Notifications in Airflow
Methods for managing notifications in your Airflow DAGs.
Introduction to Apache Airflow
Everything you need to know to get started with Apache Airflow.
Managing Dependencies in Apache Airflow
An overview of dependencies and triggers in Airflow.