AwsBatchClientHook

Amazon

A client for AWS batch services.

View on GitHub

Last Updated: Jun. 28, 2021

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

max_retriesOptional[int]exponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None
status_retriesOptional[int]number of HTTP retries to get job status, 10; polling is only used when waiters is None

Documentation

A client for AWS batch services.

Note

Several methods use a default random delay to check or poll for job status, i.e. random.uniform(DEFAULT_DELAY_MIN, DEFAULT_DELAY_MAX) Using a random interval helps to avoid AWS API throttle limits when many concurrent tasks request job-descriptions.

To modify the global defaults for the range of jitter allowed when a random delay is used to check batch job status, modify these defaults, e.g.: .. code-block:

AwsBatchClient.DEFAULT_DELAY_MIN = 0
AwsBatchClient.DEFAULT_DELAY_MAX = 5

When explicit delay values are used, a 1 second random jitter is applied to the delay (e.g. a delay of 0 sec will be a random.uniform(0, 1) delay. It is generally recommended that random jitter is added to API requests. A convenience method is provided for this, e.g. to get a random delay of 10 sec +/- 5 sec: delay = AwsBatchClient.add_jitter(10, width=5, minima=0)

Was this page helpful?