GlueCatalogPartitionSensor

Amazon

Waits for a partition to show up in AWS Glue Catalog.

View on GitHub

Last Updated: Jan. 20, 2022

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

table_nameRequiredThe name of the table to wait for, supports the dot notation (my_database.my_table)
expressionThe partition clause to wait for. This is passed as is to the AWS Glue Catalog API’s get_partitions function, and supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01". See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html #aws-glue-api-catalog-partitions-GetPartitions
aws_conn_idID of the Airflow connection where credentials and extra configuration are stored
region_nameOptional aws region name (example: us-east-1). Uses region from connection if not specified.
database_nameThe name of the catalog database where the partitions reside.
poke_intervalTime in seconds that the job should wait in between each tries

Documentation

Waits for a partition to show up in AWS Glue Catalog.

Was this page helpful?