DockerSwarmOperator

Docker

Execute a command as an ephemeral docker swarm service. Example use-case - Using Docker Swarm orchestration to make one-time scripts highly available.

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Docker provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

imagestrDocker image from which to create the container. If image tag is omitted, "latest" will be used.
api_versionstrRemote API version. Set to auto to automatically detect the server's version.
auto_removeboolAuto-removal of the container on daemon side when the container's process exits. The default is False.
commandstr or listCommand to be run in the container. (templated)
docker_urlstrURL of the host running the docker daemon. Default is unix://var/run/docker.sock
environmentdictEnvironment variables to set in the container. (templated)
force_pullboolPull the docker image on every run. Default is False.
mem_limitfloat or strMaximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like 128m or 1g.
tls_ca_certstrPath to a PEM-encoded certificate authority to secure the docker connection.
tls_client_certstrPath to the PEM-encoded certificate used to authenticate docker client.
tls_client_keystrPath to the PEM-encoded key used to authenticate docker client.
tls_hostnamestr or boolHostname to match against the docker server certificate or False to disable the check.
tls_ssl_versionstrVersion of SSL to use when communicating with docker daemon.
tmp_dirstrMount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable AIRFLOW_TMP_DIR inside the container.
userint or strDefault user inside the docker container.
docker_conn_idstrID of the Airflow connection to use
ttyboolAllocate pseudo-TTY to the container of this service This needs to be set see logs of the Docker container / service.
enable_loggingboolShow the application's logs in operator's logs. Supported only if the Docker engine is using json-file or journald logging drivers. The tty parameter should be set to use this with Python applications.

Documentation

Execute a command as an ephemeral docker swarm service. Example use-case - Using Docker Swarm orchestration to make one-time scripts highly available.

A temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. The path to the mounted directory can be accessed via the environment variable AIRFLOW_TMP_DIR.

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

Example DAGs

Was this page helpful?