AzureDataFactoryRunPipelineOperator

Microsoft Azure

Executes a data factory pipeline.

View on GitHub

Last Updated: Dec. 6, 2021

Access Instructions

Install the Microsoft Azure provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

azure_data_factory_conn_idstrThe connection identifier for connecting to Azure Data Factory.
pipeline_nameRequiredstrThe name of the pipeline to execute.
wait_for_terminationboolFlag to wait on a pipeline run’s termination. By default, this feature is enabled but could be disabled to perform an asynchronous wait for a long-running pipeline execution using the AzureDataFactoryPipelineRunSensor.
resource_group_namestrThe resource group name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the resource group name provided in the corresponding connection.
factory_namestrThe data factory name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the factory name name provided in the corresponding connection.
reference_pipeline_run_idstrThe pipeline run identifier. If this run ID is specified the parameters of the specified run will be used to create a new run.
is_recoveryboolRecovery mode flag. If recovery mode is set to True, the specified referenced pipeline run and the new run will be grouped under the same groupId.
start_activity_namestrIn recovery mode, the rerun will start from this activity. If not specified, all activities will run.
start_from_failureboolIn recovery mode, if set to true, the rerun will start from failed activities. The property will be used only if start_activity_name is not specified.
parametersDict[str, Any]Parameters of the pipeline run. These parameters are referenced in a pipeline via @pipeline().parameters.parameterName and will be used only if the reference_pipeline_run_id is not specified.
timeoutintTime in seconds to wait for a pipeline to reach a terminal status for non-asynchronous waits. Used only if wait_for_termination is True.
check_intervalintTime in seconds to check on a pipeline run’s status for non-asynchronous waits. Used only if wait_for_termination is True.

Documentation

Executes a data factory pipeline.

See also

For more information on how to use this operator, take a look at the guide: AzureDataFactoryRunPipelineOperator

Was this page helpful?