Initiate a SageMaker processing job.

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.


configdictThe configuration necessary to start a processing job (templated).For details of the configuration parameter see :py:meth:`SageMaker.Client.create_processing_job`No role entry for "py:meth" in module "docutils.parsers.rst.languages.en". Trying "py:meth" as canonical role name.Unknown interpreted text role "py:meth".
aws_conn_idstrThe AWS connection ID to use.
wait_for_completionboolIf wait is set to True, the time interval, in seconds, that the operation waits to check the status of the processing job.
print_logboolif the operator should print the cloudwatch log during processing
check_intervalintif wait is set to be true, this is the time interval in seconds which the operator will check the status of the processing job
max_ingestion_timeintIf wait is set to True, the operation fails if the processing job doesn't finish within max_ingestion_time seconds. If you set this parameter to None, the operation does not timeout.
action_if_job_existsstrBehaviour if the job name already exists. Possible options are "increment" (default) and "fail".


Initiate a SageMaker processing job.

This operator returns The ARN of the processing job created in Amazon SageMaker.

Example DAGs

Improve this module by creating an example DAG.

View Source
  1. Add an `example_dags` directory to the top-level source of the provider package with an empty `` file.
  2. Add your DAG to this directory. Be sure to include a well-written and descriptive docstring
  3. Create a pull request against the source code. Once the package gets released, your DAG will show up on the Registry.

Was this page helpful?