GlueJobHook
AmazonInteract with AWS Glue - create job, trigger, crawler
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
s3_bucketS3 bucket where logs and local etl script will be uploaded
job_nameunique job name per AWS account
descjob description
concurrent_run_limitThe maximum number of concurrent runs allowed for a job
script_locationpath to etl script on s3
retry_limitMaximum number of times to retry this job if it fails
num_of_dpusNumber of AWS Glue DPUs to allocate to this Job
region_nameaws region name (example: us-east-1)
iam_role_nameAWS IAM Role for Glue Job Execution
create_job_kwargsExtra arguments for Glue Job Creation
Documentation
Interact with AWS Glue - create job, trigger, crawler