An Apache Airflow provider for all Google services. Contains namespaces and modules for a variety of services, including Google Cloud, Google Ads, and GSuite.
Quick Install
Available Modules
Modules are python callables available from this provider package.
Types:
HooksOperatorsTransfersLogSecretsSensorsHook for the Google Ads API.
Saves list of customers on GCS in form of a csv file.
Fetches the daily results from the Google Ads API for 1-n clients Converts and saves the data as a temporary CSV file Uploads the CSV to Google Cloud …
Google Cloud AutoML hook.
Interact with BigQuery. This hook uses the Google Cloud connection.
Hook for Google Bigquery Transfer API.
Hook for Google Cloud Bigtable APIs.
Hook for the Google Cloud Build APIs.
Hook for Google Cloud Memorystore APIs.
Hook for Google Cloud Memorystore for Memcached service APIs.
Hook to connect to a remote instance in compute engine
Hook for Google Dataflow.
Hook for Google DataFusion.
Hook for connection with Dataprep API. To get connection Dataprep with Airflow you need Dataprep token. https://clouddataprep.com/documentation/api#se…
Interact with Google Cloud Datastore. This hook uses the Google Cloud connection.
Hook for Google Cloud Data Loss Prevention (DLP) APIs. Cloud DLP allows clients to detect the presence of Personally Identifiable Information (PII) an…
Hook for the Google Cloud Functions APIs.
Interact with Google Cloud Deployment Manager using the Google Cloud connection. This allows for scheduled and programmatic inspection and deletion fo…
Hook for Google Cloud Key Management service.
Hook for Google Kubernetes Engine APIs.
Hook for Google ML Engine APIs.
Hook for Google Cloud Natural Language Service.
Hook for accessing Google Pub/Sub.
Hook for Google Cloud Speech API.
Stackdriver Hook for connecting with Google Cloud Stackdriver
Hook for Google Cloud Tasks APIs. Cloud Tasks allows developers to manage the execution of background work in their applications.
Hook for Google Cloud Text to Speech API.
Hook for Google Cloud translate APIs.
Hook for Google Cloud Video Intelligence APIs.
GCSTaskHandler is a python log handler that handles and reads task instance logs. It extends airflow FileTaskHandler and uploads to and reads from GCS…
Handler that directly makes Stackdriver logging API calls.
Perform a batch prediction on Google Cloud AutoML.
Deletes a dataset and all of its contents.
Delete Google Cloud AutoML model.
Get Google Cloud AutoML model.
Lists AutoML Datasets in project.
Lists column specs in a table.
Updates a dataset.
Creates Google Cloud AutoML model.
Performs checks against BigQuery. The BigQueryCheckOperator expects a sql query that will return a single row. Each value on that first row is evaluat…
Creates a new external table in the dataset with the data from Google Cloud Storage.
This operator deletes an existing dataset from your Project in Big query. https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/delete
Deletes BigQuery tables
Executes BigQuery SQL queries in a specific BigQuery database. This operator does not assert idempotency.
Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and returns data in a python list. The number of elements in th…
This operator retrieves the list of tables in the specified dataset.
Executes a BigQuery job. Waits for the job to complete and returns job id. This operator work in the following way:
Checks that the values of metrics given as SQL expressions are within a certain tolerance of the ones from days_back before.
This operator is used to patch dataset for your Project in BigQuery. It only replaces fields that are provided in the submitted dataset resource.
This operator is used to update dataset for your Project in BigQuery. Use fields to specify which fields of dataset to update. If a field is listed in…
Upsert BigQuery table
Performs a simple value check using sql code.
Creates a new data transfer configuration.
Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the ru…
Creates a new Cloud Bigtable instance. If the Cloud Bigtable instance with the given ID exists, the operator does not compare its configuration and im…
Creates the table in the Cloud Bigtable instance.
Deletes the Cloud Bigtable instance, including its clusters and all related tables.
Updates a Cloud Bigtable cluster.
Updates an existing Cloud Bigtable instance.
Starts a build with the specified configuration.
Creates a Redis instance based on the specified tier and memory size and import a Redis RDB snapshot file from Cloud Storage into a this instance.
Creates a Redis instance based on the specified tier and memory size.
Deletes a specific Redis instance. Instance stops serving and data is deleted.
Export Redis instance data into a Redis RDB format file in Cloud Storage. In next step, deletes a this instance.
Export Redis instance data into a Redis RDB format file in Cloud Storage.
Initiates a failover of the master node to current replica node for a specific STANDARD tier Cloud Memorystore for Redis instance.
Gets the details of a specific Redis instance.
Lists all Redis instances owned by a project in either the specified location (region) or all locations.
Will update current set of Parameters to the set of specified nodes of the Memcached Instance.
Creates a Memcached instance based on the specified tier and memory size.
Deletes a specific Memcached instance. Instance stops serving and data is deleted.
Gets the details of a specific Memcached instance.
locations.
Updates the metadata and configuration of a specific Redis instance.
Updates the metadata and configuration of a specific Redis instance.
Cancels a transfer operation in Google Storage Transfer Service.
Creates a transfer job that runs periodically.
Lists long-running operations in Google Storage Transfer Service that match the specified filter.
Pauses a transfer operation in Google Storage Transfer Service.
Resumes a transfer operation in Google Storage Transfer Service.
Synchronizes an S3 bucket with a Google Cloud Storage bucket using the Google Cloud Storage Transfer Service.
Updates a transfer job that runs periodically.
Abstract base operator for Google Compute Engine operators to inherit from.
Patches the Instance Group Manager, replacing source template URL with the destination one. API V1 does not have update/patch operations for Instance …
the request.
Starts an instance in Google Compute Engine.
Creates an EntryGroup.
Creates an entry.
Creates a tag on an entry.
Creates a tag template.
Deletes an EntryGroup.
Deletes a tag.
Deletes a field in a tag template and all uses of that field.
Deletes a tag template and all tags using the template.
Gets an entry group.
Gets an entry.
Gets a tag template.
Lists the tags on an Entry.
Get an entry by target resource name.
Renames a field in a tag template.
Searches Data Catalog for multiple resources like entries, tags that match a query.
Updates an existing entry.
Updates an existing tag.
Updates a field in a tag template. This method cannot be used to update the field type.
Updates a tag template.
Start a Java Cloud DataFlow batch job. The parameters of the operation will be passed to the job.
Launching Cloud Dataflow jobs written in python. Note that both dataflow_default_options and options will be merged to specify pipeline execution para…
Starts Dataflow SQL query.
Start a Templated Cloud DataFlow job. The parameters of the operation will be passed to the job.
Creates a new Data Fusion instance in the specified project and location.
Creates a Cloud Data Fusion pipeline.
Deletes a single Date Fusion instance.
Deletes a Cloud Data Fusion pipeline.
Gets details of a single Data Fusion instance.
Lists Cloud Data Fusion pipelines.
Restart a single Data Fusion instance. At the end of an operation instance is fully restarted.
Starts a Cloud Data Fusion pipeline. Works for both batch and stream pipelines.
Stops a Cloud Data Fusion pipeline. Works for both batch and stream pipelines.
Updates a single Data Fusion instance.
Get information about the batch jobs within a Cloud Dataprep job. API documentation https://clouddataprep.com/documentation/api#section/Overview
Instantiate a WorkflowTemplate Inline on Google Cloud Dataproc. The operator will wait until the WorkflowTemplate is finished executing.
Instantiate a WorkflowTemplate on Google Cloud Dataproc. The operator will wait until the WorkflowTemplate is finished executing.
The base class for operators that launch job on DataProc.
Start a Hadoop Job on a Cloud DataProc cluster.
Start a Hive query Job on a Cloud DataProc cluster.
Submits a job to a cluster.
Start a Pig query Job on a Cloud DataProc cluster. The parameters of the operation will be passed to the cluster.
Start a PySpark Job on a Cloud DataProc cluster.
Start a Spark Job on a Cloud DataProc cluster.
Start a Spark SQL query Job on a Cloud DataProc cluster.
Updates a cluster in a project.
Allocate IDs for incomplete keys. Return list of keys.
Begins a new transaction. Returns a transaction handle.
Commit a transaction, optionally creating, deleting or modifying some entities.
Export entities from Google Cloud Datastore to Cloud Storage
Gets the latest state of a long-running operation.
Import entities from Cloud Storage to Google Cloud Datastore
Roll back a transaction.
Run a query for entities. Returns the batch of query results.
Starts asynchronous cancellation on a long-running DlpJob.
Creates a new job to inspect storage or calculate risk metrics.
Creates a DeidentifyTemplate for re-using frequently used configuration for de-identifying content, images, and storage.
Creates an InspectTemplate for re-using frequently used configuration for inspecting content, images, and storage.
Creates a job trigger to run DLP actions such as scanning storage for sensitive information on a set schedule.
Creates a pre-built stored infoType to be used for inspection.
De-identifies potentially sensitive info from a ContentItem. This method has limits on input size and output size.
Deletes a long-running DlpJob. This method indicates that the client is no longer interested in the DlpJob result. The job will be cancelled if possib…
Deletes a DeidentifyTemplate.
Deletes an InspectTemplate.
Deletes a job trigger.
Deletes a stored infoType.
Gets the latest state of a long-running DlpJob.
Gets a DeidentifyTemplate.
Gets an InspectTemplate.
Gets a stored infoType.
Finds potentially sensitive info in content. This method has limits on input size, processing time, and output size.
Lists DlpJobs that match the specified filter in the request.
Lists DeidentifyTemplates.
Returns a list of the sensitive information types that the DLP API supports.
Lists InspectTemplates.
Lists job triggers.
Lists stored infoTypes.
Redacts potentially sensitive info from an image. This method has limits on input size, processing time, and output size.
Updates the DeidentifyTemplate.
Updates the InspectTemplate.
Updates a job trigger.
Deletes the specified function from Google Cloud Functions.
Creates a function in Google Cloud Functions. If a function with this name already exists, it will be updated.
Creates a new ACL entry on the specified bucket.
Creates a new bucket. Google Cloud Storage uses a flat namespace, so you can’t create a bucket with a name that is already in use.
Deletes bucket from a Google Cloud Storage.
Deletes objects from a Google Cloud Storage bucket, either from an explicit list of object names or all objects matching a prefix.
Copies data from a source GCS location to a temporary location on the local filesystem. Runs a transformation on this file as specified by the transfo…
List all objects from the bucket with the give string prefix and delimiter in name.
Creates a new ACL entry on the specified object.
Synchronizes the contents of the buckets or bucket’s directories in the Google Cloud Services.
Create a Google Kubernetes Engine Cluster of specified dimensions The operator will wait until the cluster is created.
Deletes the cluster, including the Kubernetes endpoint and all worker nodes.
Executes a task in a Kubernetes pod in the specified Google Kubernetes Engine cluster
Runs a Life Sciences Pipeline
Creates a new model.
Creates a new version in the model
Deletes a model.
Deletes the version from the model.
Gets a particular model
Lists all available versions of the model
Operator for managing a Google Cloud ML Engine model.
Operator for managing a Google Cloud ML Engine version.
Sets a version in the model.
Operator for cleaning up failed MLEngine training job.
Finds named entities in the text along with entity types, salience, mentions for each entity, and other properties.
Finds entities, similar to AnalyzeEntities in the text and analyzes sentiment associated with each entity and its mentions.
Analyzes the sentiment of the provided text.
Classifies a document into categories.
Create a PubSub subscription.
Create a PubSub topic.
Delete a PubSub subscription.
If the queue is empty, returns empty list - never waits for messages. If you do need to wait, please use airflow.providers.google.cloud.sensors.PubSub…
Deletes a Cloud Spanner database.
Deletes a Cloud Spanner instance. If an instance does not exist, no action is taken and the operator succeeds.
Creates a new Cloud Spanner database, or if database exists, the operator does nothing.
Creates a new Cloud Spanner instance, or if an instance with the same instance_id exists in the specified project, updates the Cloud Spanner instance.
Executes an arbitrary DML query (INSERT, UPDATE, DELETE).
Updates a Cloud Spanner database with the specified DDL statement.
Recognizes speech from audio file and returns it as text.
Deletes an alerting policy.
Disables one or more enabled alerting policies identified by filter parameter. Inoperative in case the policy is already disabled.
Disables one or more enabled notification channels identified by filter parameter. Inoperative in case the policy is already disabled.
Fetches all the Alert Policies identified by the filter passed as filter parameter. The desired return type can be specified by the format parameter, …
Fetches all the Notification Channels identified by the filter passed as filter parameter. The desired return type can be specified by the format para…
Creates a new alert or updates an existing policy identified the name field in the alerts parameter.
Creates a new notification or updates an existing notification channel identified the name field in the alerts parameter.
Creates a queue in Cloud Tasks.
Deletes a queue from Cloud Tasks, even if it has tasks in it.
Gets a queue from Cloud Tasks.
Pauses a queue in Cloud Tasks.
Purges a queue by deleting all of its tasks from Cloud Tasks.
Updates a queue in Cloud Tasks.
Lists queues from Cloud Tasks.
Creates a task in Cloud Tasks.
Deletes a task from Cloud Tasks.
Gets a task from Cloud Tasks.
Forces to run a task in Cloud Tasks.
Lists the tasks in Cloud Tasks.
Translate a string or list of strings.
Recognizes speech in audio input and translates it.
Performs video annotation, annotating video labels.
Performs video annotation, annotating video shots.
Adds a Product to the specified ProductSet. If the Product is already present, no change is made.
Creates and returns a new product resource.
Creates a new ProductSet resource.
Permanently deletes a product and its reference images.
Permanently deletes a ProductSet. Products and ReferenceImages in the ProductSet are not deleted. The actual image files are not deleted from Google C…
Deletes a ReferenceImage ID resource.
Detects Document Text in the image
Detects Text in the image
Gets information associated with a Product.
Gets information associated with a ProductSet.
Run image detection and annotation for an image or a batch of images.
Removes a Product from the specified ProductSet.
Detects Document Text in the image
Makes changes to a Product resource. Only the display_name, description, and labels fields can be updated right now.
Retrieves Connection object from Google Cloud Secrets Manager
Checks for the existence of a partition within a table in Google Bigquery.
Waits for Data Transfer Service run to complete.
Sensor that waits for Cloud Bigtable table to be fully replicated to its clusters. No exception will be raised if the instance or the table does not e…
Waits for at least one operation belonging to the job to have the expected status.
Checks for the job message in Google Cloud Dataflow.
Checks for the status of a job in Google Cloud Dataflow.
Check for the state of a previously submitted Dataproc job.
Checks if an object is updated in Google Cloud Storage.
Checks for the existence of GCS objects at a given prefix, passing matches via XCom.
Checks for changes in the number of objects at prefix in Google Cloud Storage bucket and returns True if the inactivity period has passed with no incr…
Always waits for at least one message to be returned from the subscription.
Synchronizes an Azure Data Lake Storage path with a GCS bucket
Synchronizes a Azure FileShare directory content (excluding subdirectories), possibly filtered by a prefix, with a Google Cloud Storage destination pa…
Copies data from one BigQuery table to another.
Transfers a BigQuery table to a Google Cloud Storage bucket.
Copy data from Cassandra to Google Cloud Storage in JSON format
Fetches the results from the Facebook Ads API as desired in the params Converts and saves the data as a temporary JSON file Uploads the JSON to Google…
Loads files from Google Cloud Storage into BigQuery.
Copies objects from a bucket to another, with renaming if requested.
Downloads a file from Google Cloud Storage.
Transfer files from a Google Cloud Storage bucket to SFTP server.
Uploads a file or list of files to Google Cloud Storage. Optionally can compress the file for upload.
The SQL to execute.
Copy data from Postgres to Google Cloud Storage in JSON or CSV format.
Copy data from PrestoDB to Google Cloud Storage in JSON or CSV format.
Synchronizes an S3 key, possibly a prefix, with a Google Cloud Storage destination path.
Submits Salesforce query and uploads results to Google Cloud Storage
Transfer files to Google Cloud Storage from SFTP server.
Writes Google Sheet data into Google Cloud Storage.
Hook for the Google Firestore APIs.
Exports a copy of all or a subset of documents from Google Cloud Firestore to another storage system, such as Google Cloud Storage.
Hook for Google Campaign Manager.
Hook for Google Display & Video 360.
Hook for Google Search Ads 360.
Take a file from Cloud Storage and uploads it to GA via data import API.
Deletes previous GA uploads to leave the latest file to control the size of the Data Set Quota.
Returns a web property-Google Ads link to which the user has access.
Lists all accounts to which the user has access.
GA has a very particular naming convention for Data Import. Ability to prefix “ga:” to all column headers and also a dict to rename columns to match t…
Lists webProperty-Google Ads links for a given web property
Inserts conversions.
Updates existing conversions.
Deletes a report by its ID.
Retrieves a report and uploads it to GCS bucket.
Runs a report.
Creates a query.
Creates SDF operation task.
Deletes a stored query as well as the associated stored reports.
Retrieves a stored query.
Runs a stored query to generate a report.
Download SDF media and save it in the Google Cloud Storage.
Uploads line items in CSV format.
Downloads a report to GCS bucket.
Inserts a report request into the reporting system.
Check if report is ready.
Sensor for detecting the completion of SDF operation.
Sensor for detecting the completion of DV360 reports.
Polls for the status of a report request.
Hook for the Google Drive APIs.
Creates a new spreadsheet.
Copies objects from a Google Cloud Storage service to a Google Drive service, with renaming if requested.
Uploads .csv file from Google Cloud Storage to provided Google Spreadsheet.
Creates a new database inside a Cloud SQL instance.
Deletes a database from a Cloud SQL instance.
Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.
Updates settings of a Cloud SQL instance.
Updates a resource containing information about a database inside a Cloud SQL instance using patch semantics. See: https://cloud.google.com/sql/docs/m…
Copy data from MySQL to Google Cloud Storage in JSON or CSV format.
Serves DB connection configuration for Google Cloud SQL (Connections of gcpcloudsqldb:// type).
Hook for Google Cloud SQL APIs.
Abstract base operator for Google Cloud SQL operators to inherit from.
Delete a transfer job. This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collecti…
Stops an instance in Google Compute Engine.
Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and insert that data into a MySQL table.
A hook to use the Google API Discovery Service.
in JSON or CSV format.
Hook for Google Storage Transfer Service.
Hook for Google Compute Engine APIs.
Hook for Google OS login APIs.
Hook for Google Cloud Data Catalog Service.
Hook for Google Cloud Spanner APIs.
Hook for Google Cloud Vision APIs.
Creates a Google Cloud AutoML dataset.
Deploys a model. If a model is already deployed, deploying it with the same parameters has no effect. Deploying with different parameters (as e.g. cha…
Imports data to a Google Cloud AutoML dataset.
Runs prediction operation on Google Cloud AutoML.
Lists table specs in a dataset.
This operator is used to create new dataset for your Project in BigQuery. https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
Creates a new, empty table in the specified BigQuery dataset, optionally with schema.
This operator is used to return the dataset specified by dataset_id.
Deletes transfer configuration.
Deletes the Cloud Bigtable table.
Import a Redis RDB snapshot file from Cloud Storage into a Redis instance.
Updates the metadata and configuration of a specific Memcached instance.
parameters, it must be followed by apply_parameters to apply the parameters to nodes of the Memcached Instance.
Creates a new Cloud SQL instance. If an instance with the same name exists, no action will be taken and the operator will succeed.
Deletes a Cloud SQL instance.
Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage.
Copies objects from a bucket to another using the Google Cloud Storage Transfer Service.
Gets the latest state of a long-running operation in Google Storage Transfer Service.
Copies the instance template, applying specified changes.
Deletes an existing entry.
Starts flex templates with the Dataflow pipeline.
Get the specified job group. A job group is a job that is executed from a specific node in a flow. API documentation https://clouddataprep.com/documen…
Create a jobGroup, which launches the specified job as the authenticated user. This performs the same action as clicking on the Run Job button in the …
Scale, up or down, a cluster on Google Cloud Dataproc. The operator will wait until the cluster is re-scaled.
Deletes the long-running operation.
Gets a job trigger.
Re-identifies content that has been de-identified.
Updates the stored infoType by creating a new version.
Invokes a deployed Cloud Function. To be used for testing purposes as very limited traffic is allowed.
Start a Google Cloud ML Engine prediction job.
Operator for launching a MLEngine training job.
Delete a PubSub topic.
Publish messages to a PubSub topic.
Deletes a notification channel.
Enables one or more disabled alerting policies identified by filter parameter. Inoperative in case the policy is already enabled.
Enables one or more disabled alerting policies identified by filter parameter. Inoperative in case the policy is already enabled.
Resumes a queue in Cloud Tasks.
Synthesizes text to speech and stores it in Google Cloud Storage
Performs video annotation, annotating explicit content.
Creates and returns a new ReferenceImage ID resource.
Makes changes to a ProductSet resource. Only display_name can be updated currently.
Checks for the existence of a table in Google Bigquery.
Checks for the job autoscaling event in Google Cloud Dataflow.
Checks the metrics of a job in Google Cloud Dataflow.
Checks for the existence of a file in Google Cloud Storage.
Hook for Google Analytics 360.
Retrieves line items in CSV format.
Creates a report.
Detects Document Text in the image
A base hook for Google cloud-related hooks. Google cloud has a shared REST API client that is built in the same way no matter which service you use. T…
Interact with Google Sheets via Google Cloud connection Reading and writing cells in Google Sheet: https://developers.google.com/sheets/api/guides/val…
Interact with Google Cloud Storage. This hook uses the Google Cloud connection.
Creates a field in a tag template.
Hook for the Google Cloud Life Sciences APIs.
Performs DML or DDL query on an existing Cloud Sql instance. It optionally uses cloud-sql-proxy to establish secure connection with the database.
Checks for the existence of a file in Google Cloud Storage.
This operator is used to update table for your Project in BigQuery. Use fields to specify which fields of table to update. If a field is listed in fie…
Determines a list of objects that were added or modified at a GCS source location during a specific time-span, copies them to a temporary location on …
Checks for the existence of GCS objects at a given prefix, passing matches via XCom.
Writes a Google Drive file into Google Cloud Storage.
Writes a Google Drive file into local Storage.
Copy data from Oracle to Google Cloud Storage in JSON or CSV format.
Hook for the Google Secret Manager API.