DataflowStartSqlJobOperator

Google

Starts Dataflow SQL query.

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Google provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

job_namestrThe unique name to assign to the Cloud Dataflow job.
querystrThe SQL query to execute.
optionsdictJob parameters to be executed. It can be a dictionary with the following keys.For more information, look at: https://cloud.google.com/sdk/gcloud/reference/beta/dataflow/sql/query command reference
locationstrThe location of the Dataflow job (for example europe-west1)
project_idOptional[str]The ID of the GCP project that owns the job. If set to None or missing, the default project_id from the GCP connection is used.
gcp_conn_idstrThe connection ID to use connecting to Google Cloud Platform.
delegate_tostrThe account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
drain_pipelineboolOptional, set to True if want to stop streaming job by draining it instead of canceling during killing task instance. See: https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline

Documentation

Starts Dataflow SQL query.

Was this page helpful?