Apache Airflow Provider - Astronomer Providers

A library of Apache Airflow providers containing Deferrable Operators & Sensors maintained by Astronomer.

Version
1.14.0
Downloads
14,628/month
Last Published
Jan. 9, 2023
Quick Install

Available Modules

Modules are Python callables available from this provider package.

Types:

HooksTriggersSensorsOperators
GoogleBaseHookAsync
Astronomer Providers

GoogleBaseHookAsync inherits from BaseHook class, run on the trigger worker

Hooks
SnowflakeTrigger
Astronomer Providers

Snowflake Trigger inherits from the BaseTrigger,it is fired as deferred class with params to run the task in trigger worker and fetch the status for t…

Triggers
AzureDataFactoryHookAsync
Astronomer Providers

An Async Hook connects to Azure DataFactory to perform pipeline operations

Hooks
HttpSensorAsync
Astronomer Providers

Executes a HTTP GET statement and returns False on failure caused by 404 Not Found or response_check returning False.

Sensors
GCSPrefixBlobTrigger
Astronomer Providers

A trigger that fires and it looks for all the objects in the given bucket which matches the given prefix if not found sleep for certain interval and c…

Triggers
GCSUploadSessionTrigger
Astronomer Providers

Checks for changes in the number of objects at prefix in Google Cloud Storage bucket and returns Trigger Event if the inactivity period has passed wit…

Triggers
HttpTrigger
Astronomer Providers

A trigger that fires when the request to a URL returns a non-404 status code

Triggers
HttpHookAsync
Astronomer Providers

Interact with HTTP servers using Python Async.

Hooks
SnowflakeOperatorAsync
Astronomer Providers

Executes SQL code in a Snowflake database

Operators
AzureDataFactoryTrigger
Astronomer Providers

AzureDataFactoryTrigger is triggered when Azure data factory pipeline job succeeded or failed. When wait_for_termination is set to False it triggered …

Triggers
GCSUploadSessionCompleteSensorAsync
Astronomer Providers

Checks for changes in the number of objects at prefix in Google Cloud Storage bucket and returns True if the inactivity period has passed with no incr…

Sensors
SnowflakeHookAsync
Astronomer Providers

A client to interact with Snowflake.

Hooks
AzureDataFactoryRunPipelineOperatorAsync
Astronomer Providers

Executes a data factory pipeline asynchronously.

Operators
AzureDataFactoryPipelineRunStatusSensorAsync
Astronomer Providers

Checks the status of a pipeline run.

Sensors
GCSObjectExistenceSensorAsync
Astronomer Providers

Checks for the existence of a file in Google Cloud Storage.

Sensors
GCSBlobTrigger
Astronomer Providers

A trigger that fires and it finds the requested file or folder present in the given bucket.

Triggers
GCSObjectUpdateSensorAsync
Astronomer Providers

Async version to check if an object is updated in Google Cloud Storage

Sensors
BigQueryIntervalCheckTrigger
Astronomer Providers

BigQueryIntervalCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class

Triggers
GCSCheckBlobUpdateTimeTrigger
Astronomer Providers

A trigger that makes an async call to GCS to check whether the object is updated in a bucket.

Triggers
BigQueryTableHookAsync
Astronomer Providers

Class to get async hook for Bigquery Table Async

Hooks
DataProcSubmitTrigger
Astronomer Providers

Check for the state of a previously submitted Dataproc job.

Triggers
DataprocHookAsync
Astronomer Providers

Async Hook for Google Cloud Dataproc APIs.

Hooks
GCSObjectsWithPrefixExistenceSensorAsync
Astronomer Providers

Async Operator that Checks for the existence of GCS objects at a given prefix, passing matches via XCom.

Sensors
BigQueryValueCheckTrigger
Astronomer Providers

BigQueryValueCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class

Triggers
GCSHookAsync
Astronomer Providers

GCSHookAsync run on the trigger worker, inherits from GoogleBaseHookAsync

Hooks
DataprocSubmitJobOperatorAsync
Astronomer Providers

Submits a job to a cluster and wait until is completely finished or any error occurs.

Operators
BigQueryInsertJobTrigger
Astronomer Providers

BigQueryInsertJobTrigger run on the trigger worker to perform insert operation

Triggers
BigQueryValueCheckOperatorAsync
Astronomer Providers

Operators
BigQueryTableExistenceSensorAsync
Astronomer Providers

Checks for the existence of a table in Google Big Query.

Sensors
BigQueryCheckTrigger
Astronomer Providers

BigQueryCheckTrigger run on the trigger worker

Triggers
BigQueryCheckOperatorAsync
Astronomer Providers

BigQueryCheckOperatorAsync is asynchronous operator, submit the job and check for the status in async mode by using the job id

Operators
BigQueryGetDataTrigger
Astronomer Providers

BigQueryGetDataTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class

Triggers
BigQueryIntervalCheckOperatorAsync
Astronomer Providers

Checks asynchronously that the values of metrics given as SQL expressions are within a certain tolerance of the ones from days_back before.

Operators
BigQueryGetDataOperatorAsync
Astronomer Providers

Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and returns data in a python list. The number of elements in th…

Operators
DatabricksTrigger
Astronomer Providers

Wait asynchronously for databricks job to reach the terminal state.

Triggers
BigQueryInsertJobOperatorAsync
Astronomer Providers

Starts a BigQuery job asynchronously, and returns job id. This operator works in the following way:

Operators
DatabricksHookAsync
Astronomer Providers

Interact with Databricks.

Hooks
TaskStateTrigger
Astronomer Providers

Waits asynchronously for a task in a different DAG to complete for a specific logical date.

Triggers
FileTrigger
Astronomer Providers

A trigger that fires exactly once after it finds the requested file or folder.

Triggers
KubernetesPodOperatorAsync
Astronomer Providers

Async (deferring) version of KubernetesPodOperator

Operators
FileSensorAsync
Astronomer Providers

Waits for a file or folder to land in a filesystem using async.

Sensors
DagStateTrigger
Astronomer Providers

Waits asynchronously for a task in a different DAG to complete for a specific logical date.

Triggers
ExternalTaskSensorAsync
Astronomer Providers

Sensors
DatabricksRunNowOperatorAsync
Astronomer Providers

Operators
DatabricksSubmitRunOperatorAsync
Astronomer Providers

Operators
LivyTrigger
Astronomer Providers

Check for the state of a previously submitted job with batch_id

Triggers
KubernetesHookAsync
Astronomer Providers

Hooks
WaitContainerTrigger
Astronomer Providers

First, waits for pod pod_name to reach running state within pending_phase_timeout. Next, waits for container_name to reach a terminal state.

Triggers
LivyHookAsync
Astronomer Providers

Hook for Apache Livy through the REST API using LivyHookAsync

Hooks
HivePartitionTrigger
Astronomer Providers

A trigger that fires and it looks for a partition in the given table in the database or wait for the partition.

Triggers
NamedHivePartitionTrigger
Astronomer Providers

A trigger that fires, and it looks for a partition in the given table in the database or wait for the partition.

Triggers
NamedHivePartitionSensorAsync
Astronomer Providers

Waits asynchronously for a set of partitions to show up in Hive.

Sensors
HiveCliHookAsync
Astronomer Providers

HiveCliHookAsync to interact with the Hive using impyla library

Hooks
S3PrefixTrigger
Astronomer Providers

S3PrefixTrigger class is fired as deferred class with params to run the task in trigger worker

Triggers
S3KeyTrigger
Astronomer Providers

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker

Triggers
HivePartitionSensorAsync
Astronomer Providers

Waits for a given partition to show up in Hive table asynchronously.

Sensors
S3PrefixSensorAsync
Astronomer Providers

Async implementation of the S3 Prefix Sensor. Gets deferred onto the Trigggerer and pokes for a prefix or all prefixes to exist. A prefix is the first…

Sensors
S3KeysUnchangedTrigger
Astronomer Providers

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker

Triggers
S3KeySizeSensorAsync
Astronomer Providers

Waits for a key (a file-like instance on S3) to be present and be more than some size in a S3 bucket asynchronously. S3 being a key/value it does not …

Sensors
S3KeysUnchangedSensorAsync
Astronomer Providers

Checks for changes in the number of objects at prefix in AWS S3 bucket and returns True if the inactivity period has passed with no increase in the nu…

Sensors
S3KeySizeTrigger
Astronomer Providers

S3KeySizeTrigger is fired as deferred class with params to run the task in trigger worker, S3 Objects have size more than 0

Triggers
S3KeySensorAsync
Astronomer Providers

Waits for a key (a file-like instance on S3) to be present in a S3 bucket asynchronously. S3 being a key/value it does not support folders. The path i…

Sensors
RedshiftSQLOperatorAsync
Astronomer Providers

Executes SQL Statements against an Amazon Redshift cluster”

Operators
RedshiftSQLTrigger
Astronomer Providers

RedshiftSQLTrigger is fired as deferred class with params to run the task in trigger worker

Triggers
S3HookAsync
Astronomer Providers

Interact with AWS S3, using the aiobotocore library.

Hooks
RedshiftHookAsync
Astronomer Providers

Interact with AWS Redshift using aiobotocore python library

Hooks
RedshiftSQLHookAsync
Astronomer Providers

RedshiftSQL async hook inherits from RedshiftDataHook to interact with AWS redshift cluster database

Hooks
RedshiftPauseClusterOperatorAsync
Astronomer Providers

Pause an AWS Redshift Cluster if cluster status is in available state, and Few points on the cluster creation to avoid this type of Exception ex:- ‘Yo…

Operators
RedshiftResumeClusterOperatorAsync
Astronomer Providers

Resume a paused AWS Redshift Cluster, and Few points on the cluster creation to avoid this type of Exception ex:- ‘You can’t Resume cluster redshift-c…

Operators
RedshiftDataHook
Astronomer Providers

RedshiftDataHook inherits from AwsBaseHook to connect with AWS redshift by using boto3 client_type as redshift-data we can interact with redshift clus…

Hooks
RedshiftClusterTrigger
Astronomer Providers

RedshiftClusterTrigger is fired as deferred class with params to run the task in trigger worker

Triggers
RedshiftClusterSensorTrigger
Astronomer Providers

RedshiftClusterSensorTrigger is fired as deferred class with params to run the task in trigger worker

Triggers
RedshiftClusterSensorAsync
Astronomer Providers

Waits for a Redshift cluster to reach a specific status.

Sensors
EmrJobFlowSensorTrigger
Astronomer Providers

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when EMR JobFlow is created

Triggers
EmrStepSensorTrigger
Astronomer Providers

A trigger that fires once AWS EMR cluster step reaches either target or failed state

Triggers
EmrStepSensorAsync
Astronomer Providers

Async (deferring) version of EmrStepSensor

Sensors
EmrStepSensorHookAsync
Astronomer Providers

A thin wrapper to interact with AWS EMR API

Hooks
EmrJobFlowSensorAsync
Astronomer Providers

Async EMR Job flow sensor polls for the cluster state until it reaches any of the target states. If it fails the sensor errors, failing the task. With…

Sensors
EmrContainerSensorAsync
Astronomer Providers

EmrContainerSensorAsync is async version of EmrContainerSensor, Asks for the state of the job run until it reaches a failure state or success state. I…

Sensors
EmrJobFlowHookAsync
Astronomer Providers

EmrJobFlowHookAsync is wrapper Interact with AWS EMR.Using Aiobotocore client makes API call to get cluster-level details by job_flow_id. Additional a…

Hooks
EmrContainerSensorTrigger
Astronomer Providers

The EmrContainerSensorTrigger is triggered when EMR container is created, it polls for the AWS EMR EKS Virtual Cluster Job status. It is fired as defe…

Triggers
EmrContainerOperatorAsync
Astronomer Providers

An async operator that submits jobs to EMR on EKS virtual clusters.

Operators
EmrContainerOperatorTrigger
Astronomer Providers

The EmrContainerSensorTrigger is triggered when EMR container is created, it polls for the AWS EMR EKS Virtual Cluster Job status. It is fired as defe…

Triggers
EmrContainerHookAsync
Astronomer Providers

The EmrContainerHookAsync interact with AWS EMR EKS Virtual Cluster to run, poll jobs and return job status Additional arguments (such as aws_conn_id)…

Hooks
AwsBaseHookAsync
Astronomer Providers

Interacts with AWS using aiobotocore asynchronously.

Hooks
BigQueryHookAsync
Astronomer Providers

Big query async hook inherits from GoogleBaseHookAsync class and connects to the google Big query

Hooks
LivyOperatorAsync
Astronomer Providers

This operator wraps the Apache Livy batch REST API, allowing to submit a Spark application to the underlying cluster asynchronously.

Operators
ADFPipelineRunStatusSensorTrigger
Astronomer Providers

ADFPipelineRunStatusSensorTrigger is fired as deferred class with params to run the task in trigger worker, when ADF Pipeline is running

Triggers
BigQueryTableExistenceTrigger
Astronomer Providers

Initialise the BigQuery Table Existence Trigger with needed parameters

Triggers
BatchClientHookAsync
Astronomer Providers

Async client for AWS Batch services.

Hooks
BatchOperatorAsync
Astronomer Providers

Execute a job asynchronously on AWS Batch

Operators
GKEStartPodOperatorAsync
Astronomer Providers

Executes a task in a Kubernetes pod in the specified Google Kubernetes Engine cluster

Operators
BatchOperatorTrigger
Astronomer Providers

Checks for the state of a previously submitted job to AWS Batch. BatchOperatorTrigger is fired as deferred class with params to poll the job state in …

Triggers
GKEStartPodTrigger
Astronomer Providers

Fetch GKE cluster config and wait for pod to start up.

Triggers