DockerOperator

Docker

Execute a command inside a docker container.

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Docker provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

imagestrDocker image from which to create the container. If image tag is omitted, "latest" will be used.
api_versionstrRemote API version. Set to auto to automatically detect the server's version.
commandstr or listCommand to be run in the container. (templated)
container_namestr or NoneName of the container. Optional (templated)
cpusfloatNumber of CPUs to assign to the container. This value gets multiplied with 1024. See https://docs.docker.com/engine/reference/run/#cpu-share-constraint
docker_urlstrURL of the host running the docker daemon. Default is unix://var/run/docker.sock
environmentdictEnvironment variables to set in the container. (templated)
private_environmentdictPrivate environment variables to set in the container. These are not templated, and hidden from the website.
force_pullboolPull the docker image on every run. Default is False.
mem_limitfloat or strMaximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like 128m or 1g.
host_tmp_dirstrSpecify the location of the temporary directory on the host which will be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
network_modestrNetwork mode for the container.
tls_ca_certstrPath to a PEM-encoded certificate authority to secure the docker connection.
tls_client_certstrPath to the PEM-encoded certificate used to authenticate docker client.
tls_client_keystrPath to the PEM-encoded key used to authenticate docker client.
tls_hostnamestr or boolHostname to match against the docker server certificate or False to disable the check.
tls_ssl_versionstrVersion of SSL to use when communicating with docker daemon.
tmp_dirstrMount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable AIRFLOW_TMP_DIR inside the container.
userint or strDefault user inside the docker container.
volumeslistList of volumes to mount into the container, e.g. ['/host/path:/container/path', '/host/path2:/container/path2:ro'].
working_dirstrWorking directory to set on the container (equivalent to the -w switch the docker client)
xcom_allboolPush all the stdout or just the last line. The default is False (last line).
docker_conn_idstrID of the Airflow connection to use
dnslist[str]Docker custom DNS servers
dns_searchlist[str]Docker custom DNS search domain
auto_removeboolAuto-removal of the container on daemon side when the container's process exits. The default is False.
shm_sizeintSize of /dev/shm in bytes. The size must be greater than 0. If omitted uses system default.
ttyboolAllocate pseudo-TTY to the container This needs to be set see logs of the Docker container.
privilegedboolGive extended privileges to this container.
cap_addlist[str]Include container capabilities

Documentation

Execute a command inside a docker container.

A temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. The path to the mounted directory can be accessed via the environment variable AIRFLOW_TMP_DIR.

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

Was this page helpful?