AwsBatchOperator

Amazon

Execute a job on AWS Batch

View Source

Last Updated: Apr. 2, 2021

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

job_namestrthe name for the job that will run on AWS Batch (templated)
job_definitionstrthe job definition name on AWS Batch
job_queuestrthe queue name on AWS Batch
overridesOptional[dict]the containerOverrides parameter for boto3 (templated)
array_propertiesOptional[dict]the arrayProperties parameter for boto3
parametersOptional[dict]the parameters for boto3 (templated)
job_idOptional[str]the job ID, usually unknown (None) until the submit_job operation gets the jobId defined by AWS Batch
waitersOptional[AwsBatchWaiters]an :py:class:`.AwsBatchWaiters` object (see note below); if None, polling is used with max_retries and status_retries.No role entry for "py:class" in module "docutils.parsers.rst.languages.en". Trying "py:class" as canonical role name.Unknown interpreted text role "py:class".
max_retriesintexponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None
status_retriesintnumber of HTTP retries to get job status, 10; polling is only used when waiters is None
aws_conn_idstrconnection id of AWS credentials / region name. If None, credential boto3 strategy will be used.
region_namestrregion name to use in AWS Hook. Override the region_name in connection (if provided)
tagsdictcollection of tags to apply to the AWS Batch job submission if None, no tags are submitted

Documentation

Execute a job on AWS Batch

Note

Any custom waiters must return a waiter for these calls: .. code-block:: python

waiter = waiters.get_waiter(“JobExists”) waiter = waiters.get_waiter(“JobRunning”) waiter = waiters.get_waiter(“JobComplete”)

Example DAGs

Improve this module by creating an example DAG.

View Source
  1. Add an `example_dags` directory to the top-level source of the provider package with an empty `__init__.py` file.
  2. Add your DAG to this directory. Be sure to include a well-written and descriptive docstring
  3. Create a pull request against the source code. Once the package gets released, your DAG will show up on the Registry.

Was this page helpful?