GCSTaskHandler

Google

GCSTaskHandler is a python log handler that handles and reads task instance logs. It extends airflow FileTaskHandler and uploads to and reads from GCS remote storage. Upon log reading failure, it reads from host machine’s local disk.

View Source

Last Updated: Mar. 6, 2021

Access Instructions

Install the Google provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

base_log_folderstrBase log folder to place logs.
gcs_log_folderstrPath to a remote location where logs will be saved. It must have the prefix gs://. For example: gs://bucket/remote/log/location
filename_templatestrtemplate filename string
gcp_key_pathstrPath to Google Cloud Service Account file (JSON). Mutually exclusive with gcp_keyfile_dict. If omitted, authorization based on the Application Default Credentials will be used.
gcp_keyfile_dictdictDictionary of keyfile parameters. Mutually exclusive with gcp_key_path.
gcp_scopesstrComma-separated string containing OAuth2 scopes
project_idstrProject ID to read the secrets from. If not passed, the project ID from credentials will be used.

Documentation

GCSTaskHandler is a python log handler that handles and reads task instance logs. It extends airflow FileTaskHandler and uploads to and reads from GCS remote storage. Upon log reading failure, it reads from host machine’s local disk.

Example DAGs

Improve this module by creating an example DAG.

View Source
  1. Add an `example_dags` directory to the top-level source of the provider package with an empty `__init__.py` file.
  2. Add your DAG to this directory. Be sure to include a well-written and descriptive docstring
  3. Create a pull request against the source code. Once the package gets released, your DAG will show up on the Registry.

Was this page helpful?