GlueCrawlerOperator

Amazon

Creates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.

View on GitHub

Last Updated: Jun. 22, 2022

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

configConfigurations for the AWS Glue crawler
aws_conn_idaws connection to use
poll_intervalTime (in seconds) to wait between two consecutive calls to check crawler status
wait_for_completionWhether or not wait for crawl execution completion. (default: True)

Documentation

Creates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.

See also

For more information on how to use this operator, take a look at the guide: Create an AWS Glue crawler

Was this page helpful?