Airflow Guides

Topics

AWS
Airflow UI
Astro
Azure
Basics
Best Practices
Components
Concurrency
Connections
DAGs
Data Quality
Data quality
Database
Dataset
Dependencies
ETL
Executors
Great Expectations
Hooks
Integrations
Kubernetes
Lineage
Logging
Machine Learning
Operators
Parallelism
Plugins
Resources
SQL
Scheduling
Secrets
Sensors
Soda
Start
Subdags
Task Groups
Tasks
Templating
Testing
Windows
Workers
XCom

Datasets and Data-Aware Scheduling in Airflow

Using datasets to implement DAG dependencies and scheduling in Airflow.

Soda Core and Airflow

Using Soda Core to implement data quality checks in Airflow DAGs.

Data quality and Airflow

Check the quality of your data using Airflow.

Orchestrating Snowflake Queries with Airflow

How to use Airflow to get enhanced observability and compute savings while orchestrating your Snowflake jobs.

Dynamic Tasks in Airflow

How to dynamically create tasks at runtime in your Airflow DAGs.

OpenLineage and Airflow

Using OpenLineage and Marquez to get lineage data from your Airflow DAGs.

Orchestrating Redshift Operations from Airflow

Setting up a connection to Redshift and using available Redshift modules.

The Astro Python SDK for ETL

Using the Astro Python SDK to implement ETL use cases in Airflow.

Introduction to Airflow Decorators

An overview of Airflow decorators and how they can improve the DAG authoring experience.

Deferrable Operators

How to implement deferrable operators to save cost and resources with Airflow.

Debugging DAGs

A beginner's guide to figuring out what's going wrong with your Airflow DAGs

Rerunning Airflow DAGs

How to use catchup, backfill, and cleared task instances in Airflow.

Scheduling and Timetables in Airflow

Everything you need to know about scheduling your Airflow DAGs.

Airflow Data Quality Checks with SQL Operators

Executing queries in Apache Airflow DAGs to ensure data quality.

Airflow Pools

Using pools to control task parallelism in Airflow.

Integrating Airflow and dbt

Running dbt models in your Airflow DAGs.

Using Airflow with SageMaker

Methods for orchestrating SageMaker machine learning pipelines with Airflow.

Executing Notebooks with Airflow

Methods for orchestrating commonly used notebooks with Airflow.

Cross-DAG Dependencies

How to implement dependencies between your Airflow DAGs.

Testing Airflow DAGs

How to apply test-driven development practices to your Airflow DAGs.

Using Task Groups in Airflow

Using Task Groups to build modular workflows in Airflow.

Custom XCom Backends

Creating a custom XCom backend with Airflow 2.0.

Passing Data Between Airflow Tasks

Methods for sharing metadata and information between tasks in your Apache Airflow DAGs, including XCom.

Deploying Kedro Pipelines to Apache Airflow

How to use the kedro-airflow plugin to change your Kedro pipelines into Apache Airflow DAGs and deploy them to a production environment.

Orchestrating Databricks Jobs with Airflow

Orchestrating Databricks Jobs from your Apache Airflow DAGs.

Executing Azure Data Factory Pipelines with Airflow

Triggering remote jobs in Azure Data Factory from your Apache Airflow DAGs.

Executing Azure Data Explorer Queries with Airflow

Executing Azure Data Explorer queries from your Apache Airflow DAGs.

Orchestrating Azure Container Instances with Airflow

Orchestrating containers with Azure Container Instances from your Apache Airflow DAGs.

Get Started with Apache Airflow 2.0

Test Apache Airflow 2.0 on your local machine with the Astro CLI.

Using Airflow to Execute SQL

Executing queries, parameterizing queries, and embedding SQL-driven ETL in Apache Airflow DAGs.

Integrating Airflow and Great Expectations

Using the Great Expectations provider natively in your Airflow DAGs.

Understanding the Airflow Metadata Database

An structural walkthrough of Apache Airflow's metadata database, with a full ERD.

Executing Talend Jobs with Airflow

Triggering remote jobs in Talend from your Apache Airflow DAGs.

Integrating Airflow and Hashicorp Vault

Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.

Importing Custom Hooks & Operators

How to correctly import custom hooks and operators.

Scaling Out Airflow

How to tune your Airflow environment so it scales with your DAGs.

Airflow Executors Explained

A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.

Logging in Airflow

Demystifying Airflow's logging configuration.

Best Practices Calling AWS Lambda from Airflow

A few tips, guidelines, and best practices for calling Lambda from Airflow

Using the KubernetesPodOperator

Use the KubernetesPodOperator in Airflow to run tasks in Kubernetes Pods

Using the BashOperator

Use the BashOperator in Airflow to execute bash commands and scripts

Using SubDAGs in Airflow

Using SubDAGs to build modular workflows in Airflow.

Templating in Airflow

How to leverage the power of Jinja templating when writing your DAGs.

Branching in Airflow

Use Apache Airflow's BranchPythonOperator and ShortCircuitOperator to execute conditional branches in your workflow

Airflow's Components

Learn about the core components of Apache Airflow's infrastructure.

The Airflow UI

A high-level overview of the Airflow UI

Running Airflow on Windows 10 & WSL

How to spin up Airflow on your Windows system.

Managing your Connections in Apache Airflow

An overview of how connections work in the Airflow UI.

DAG Writing Best Practices in Apache Airflow

How to create effective, clean, and functional DAGs.

Introduction to Airflow DAGs

How to write your first DAG in Apache Airflow

Error Notifications in Airflow

Methods for managing notifications in your Airflow DAGs.

Dynamically Generating DAGs in Airflow

Using a base DAG template to create multiple DAGs.

Introduction to Apache Airflow

Everything you need to know to get started with Apache Airflow.

Managing Airflow Code

Guidelines for Working with Multiple Airflow Projects

Managing Dependencies in Apache Airflow

An overview of dependencies and triggers in Airflow.

Using Apache Airflow Plugins

A crash-course in using Airflow Plugins.

Hooks 101

An introduction to Hooks in Apache Airflow.

Sensors 101

An introduction to Sensors in Apache Airflow.

Operators 101

An introduction to Operators in Apache Airflow.