load_file

Astro SDKCertified

Load CSV or Parquet files from local storage, Amazon S3, or Google Cloud Storage into a SQL database.

View on GitHub

Last Updated: Jun. 13, 2022

Access Instructions

Install the Astro SDK provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

pathRequiredstrThe path to a file.
output_tableastro.sql.table.TableThe ``Table`` to create as a result from the file contents.
file_conn_idstrThe ID of the configured Airflow Connection to use to retrieve the source file.
task_idstrThe ID of the task.

Documentation

To create an ELT pipeline, users can first load (CSV or parquet) data (from local, S3, or GCS) into a SQL database with the load_file function.

Example:
from astro.sql import load_file
from astro.sql.table import Table
raw_orders = load_file(
path="s3://my/s3/path.csv",
file_conn_id="my_s3_conn",
output_table=Table(table_name="my_table", conn_id="postgres_conn"),
)

Was this page helpful?