KubernetesPodOperator

Kubernetes

Execute a task in a Kubernetes Pod

View on GitHub

Last Updated: Jun. 16, 2021

Access Instructions

Install the Kubernetes provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

namespacestrthe namespace to run within kubernetes.
imagestrDocker image you wish to launch. Defaults to hub.docker.com, but fully qualified URLS will point to custom repositories. (templated)
namestrname of the pod in which the task will run, will be used (plus a random suffix) to generate a pod id (DNS-1123 subdomain, containing only [a-z0-9.-]).
cmdslist[str]entrypoint of the container. (templated) The docker images’s entrypoint is used if this is not provided.
argumentslist[str]arguments of the entrypoint. (templated) The docker image’s CMD is used if this is not provided.
portslist[k8s.V1ContainerPort]ports for launched pod.
volume_mountslist[k8s.V1VolumeMount]volumeMounts for launched pod.
volumeslist[k8s.V1Volume]volumes for launched pod. Includes ConfigMaps and PersistentVolumes.
env_varslist[k8s.V1EnvVar]Environment variables initialized in the container. (templated)
secretslist[airflow.kubernetes.secret.Secret]Kubernetes secrets to inject in the container. They can be exposed as environment vars or files in a volume.
in_clusterboolrun kubernetes client with in_cluster configuration.
cluster_contextstrcontext that points to kubernetes cluster. Ignored when in_cluster is True. If None, current-context is used.
reattach_on_restartboolif the scheduler dies while the pod is running, reattach and monitor
labelsdictlabels to apply to the Pod. (templated)
startup_timeout_secondsinttimeout in seconds to startup the pod.
get_logsboolget the stdout of the container as logs of the tasks.
image_pull_policystrSpecify a policy to cache or always pull an image.
annotationsdictnon-identifying metadata you can attach to the Pod. Can be a large range of data, and can include characters that are not permitted by labels.
resourcesk8s.V1ResourceRequirementsA dict containing resources requests and limits. Possible keys are request_memory, request_cpu, limit_memory, limit_cpu, and limit_gpu, which will be used to generate airflow.kubernetes.pod.Resources. See also kubernetes.io/docs/concepts/configuration/manage-compute-resources-container
affinityk8s.V1AffinityA dict containing a group of affinity scheduling rules.
config_filestrThe path to the Kubernetes config file. (templated) If not specified, default value is ~/.kube/config
node_selectorsdictA dict containing a group of scheduling rules.
image_pull_secretsList[k8s.V1LocalObjectReference]Any image pull secrets to be given to the pod. If more than one secret is required, provide a comma separated list: secret_a,secret_b
service_account_namestrName of the service account
is_delete_operator_podboolWhat to do when the pod reaches its final state, or the execution is interrupted. If False (default): do nothing, If True: delete the pod
hostnetworkboolIf True enable host networking on the pod.
tolerationsList[k8s.V1Toleration]A list of kubernetes tolerations.
security_contextdictsecurity options the pod should run with (PodSecurityContext).
dnspolicystrdnspolicy for the pod.
schedulernamestrSpecify a schedulername for the pod
full_pod_speckubernetes.client.models.V1PodThe complete podSpec
init_containerslist[kubernetes.client.models.V1Container]init container for the launched Pod
log_events_on_failureboolLog the pod’s events if a failure occurs
do_xcom_pushboolIf True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.
pod_template_filestrpath to pod template file (templated)
priority_class_namestrpriority class name for the launched Pod
termination_grace_periodintTermination grace period if task killed in UI, defaults to kubernetes default

Documentation

Execute a task in a Kubernetes Pod

See also

For more information on how to use this operator, take a look at the guide: KubernetesPodOperator

Note

If you use Google Kubernetes Engine and Airflow is not running in the same cluster, consider using GKEStartPodOperator, which simplifies the authorization process.

Was this page helpful?