load_file

AstroCertified

Load CSV or Parquet files from local storage, Amazon S3, or Google Cloud Storage into a SQL database.

View on GitHub

Last Updated: Dec. 16, 2021

Access Instructions

Install the Astro provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

pathRequiredstrThe path to a file.
output_tableastro.sql.table.TableThe ``Table`` to create as a result from the file contents.
file_conn_idstrThe ID of the configured Airflow Connection to use to retrieve the source file.
task_idstrThe ID of the task.

Documentation

To create an ELT pipeline, users can first load (CSV or parquet) data (from local, S3, or GCS) into a SQL database with the load_file function.

Example:
from astro.sql import load_file
from astro.sql.table import Table
raw_orders = load_file(
path="s3://my/s3/path.csv",
file_conn_id="my_s3_conn",
output_table=Table(table_name="my_table", conn_id="postgres_conn"),
)

Was this page helpful?