AzureDataLakeStorageListOperator

Microsoft Azure

List all files from the specified path

View Source

Last Updated: Apr. 16, 2021

Access Instructions

Install the Microsoft Azure provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

pathstrThe Azure Data Lake path to find the objects. Supports glob strings (templated)
azure_data_lake_conn_idstrThe connection ID to use when connecting to Azure Data Lake Storage.

Documentation

List all files from the specified path

This operator returns a python list with the names of files which can be used by

xcom in the downstream tasks.

Example:

The following Operator would list all the Parquet files from folder/output/ folder in the specified ADLS account

adls_files = AzureDataLakeStorageListOperator(
task_id='adls_files',
path='folder/output/*.parquet',
azure_data_lake_conn_id='azure_data_lake_default'
)

Example DAGs

Improve this module by creating an example DAG.

View Source
  1. Add an `example_dags` directory to the top-level source of the provider package with an empty `__init__.py` file.
  2. Add your DAG to this directory. Be sure to include a well-written and descriptive docstring
  3. Create a pull request against the source code. Once the package gets released, your DAG will show up on the Registry.

Was this page helpful?