SmartSensorOperator

Apache Airflow Certified

Smart sensor operators are derived from this class.

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Apache Airflow provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

soft_failboolSet to true to mark the task as SKIPPED on failure
poke_intervalintTime in seconds that the job should wait in between each tries.
smart_sensor_timeoutfloatTime, in seconds before the internal sensor job times out if poke_timeout is not defined.
shard_minintshard code lower bound (inclusive)
shard_maxintshard code upper bound (exclusive)
poke_timeoutfloatTime, in seconds before the task times out and fails.

Documentation

Smart sensor operators are derived from this class.

Smart Sensor operators keep refresh a dictionary by visiting DB. Taking qualified active sensor tasks. Different from sensor operator, Smart sensor operators poke for all sensor tasks in the dictionary at a time interval. When a criteria is met or fail by time out, it update all sensor task state in task_instance table

Example DAGs

Improve this module by creating an example DAG.

View Source
  1. Add an `example_dags` directory to the top-level source of the provider package with an empty `__init__.py` file.
  2. Add your DAG to this directory. Be sure to include a well-written and descriptive docstring
  3. Create a pull request against the source code. Once the package gets released, your DAG will show up on the Registry.

Was this page helpful?