AwsGlueCatalogPartitionSensor

Amazon

Waits for a partition to show up in AWS Glue Catalog.

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

table_namestrThe name of the table to wait for, supports the dot notation (my_database.my_table)
expressionstrThe partition clause to wait for. This is passed as is to the AWS Glue Catalog API's get_partitions function, and supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01". See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html #aws-glue-api-catalog-partitions-GetPartitions
aws_conn_idstrID of the Airflow connection where credentials and extra configuration are stored
region_namestrOptional aws region name (example: us-east-1). Uses region from connection if not specified.
database_namestrThe name of the catalog database where the partitions reside.
poke_intervalintTime in seconds that the job should wait in between each tries

Documentation

Waits for a partition to show up in AWS Glue Catalog.

Example DAGs

Improve this module by creating an example DAG.

View Source
  1. Add an `example_dags` directory to the top-level source of the provider package with an empty `__init__.py` file.
  2. Add your DAG to this directory. Be sure to include a well-written and descriptive docstring
  3. Create a pull request against the source code. Once the package gets released, your DAG will show up on the Registry.

Was this page helpful?