EmrAddStepsOperator

Amazon

An operator that adds steps to an existing EMR job_flow.

View on GitHub

Last Updated: May. 7, 2021

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

job_flow_idOptional[str]id of the JobFlow to add steps to. (templated)
job_flow_nameOptional[str]name of the JobFlow to add steps to. Use as an alternative to passing job_flow_id. will search for id of JobFlow with matching name in one of the states in param cluster_states. Exactly one cluster like this should exist or will fail. (templated)
cluster_stateslistAcceptable cluster states when searching for JobFlow id by job_flow_name. (templated)
aws_conn_idstraws connection to uses
stepslist|strboto3 style steps or reference to a steps file (must be ‘.json’) to be added to the jobflow. (templated)
do_xcom_pushboolif True, job_flow_id is pushed to XCom with key job_flow_id.

Documentation

An operator that adds steps to an existing EMR job_flow.

Was this page helpful?