GlueJobHook

Amazon

Interact with AWS Glue - create job, trigger, crawler

View on GitHub

Last Updated: Jun. 8, 2022

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

s3_bucketS3 bucket where logs and local etl script will be uploaded
job_nameunique job name per AWS account
descjob description
concurrent_run_limitThe maximum number of concurrent runs allowed for a job
script_locationpath to etl script on s3
retry_limitMaximum number of times to retry this job if it fails
num_of_dpusNumber of AWS Glue DPUs to allocate to this Job
region_nameaws region name (example: us-east-1)
iam_role_nameAWS IAM Role for Glue Job Execution
create_job_kwargsExtra arguments for Glue Job Creation

Documentation

Interact with AWS Glue - create job, trigger, crawler

Was this page helpful?