SparkKubernetesSensor

Kubernetes

Checks sparkApplication object in kubernetes cluster:

View Source

Last Updated: Apr. 27, 2021

Access Instructions

Install the Kubernetes provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

application_namestrspark Application resource name
namespacestrthe kubernetes namespace where the sparkApplication reside in
kubernetes_conn_idstrthe connection to Kubernetes cluster
attach_logbooldetermines whether logs for driver pod should be appended to the sensor log

Documentation

Checks sparkApplication object in kubernetes cluster:

See also

For more detail about Spark Application Object have a look at the reference: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.1.0-2.4.5/docs/api-docs.md#sparkapplication

Was this page helpful?