export_file

Astro SDKCertified

Export a database table as a CSV or Parquet file to local storage, Amazon S3, or Google Cloud Storage.

View on GitHub

Last Updated: Jun. 13, 2022

Access Instructions

Install the Astro SDK provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

output_file_pathRequiredstrThe target file path.
tableastro.sql.table.TableThe ``Table`` to use as a source for offloading to a file.
output_conn_idstrThe ID of the configured Airflow Connection to use to save the output file (if S3 of GCS).
output_file_formatstrThe file format to use when writing the output file. Valid values are 'csv' or 'parquet'.
overwriteboolFlag whether or not to overwrite the output file if it already exists.
task_idstrThe ID of the task.

Documentation

Export a database table as a CSV or Parquet file to local storage, Amazon S3, or Google Cloud Storage.

Example:
@dag(
default_args=default_args,
schedule_interval=None,
start_date=datetime(2021, 1, 1),
tags=["demo"],
)
def demo_with_s3_and_csv():
t1 = load_file(
path="s3://tmp9/homes.csv",
file_conn_id="my_aws_conn",
database="astro",
output_table_name="expected_table_from_s3",
)
t2 = load_file(
path="tests/data/homes.csv",
database="astro",
output_table_name="expected_table_from_csv",
)
save_file(
output_file_path="s3://tmp9/homes.csv",
table=t1,
input_conn_id="postgres_conn",
overwrite=True,
)
save_file(
output_file_path="tests/data/homes_output.csv",
table=t2,
input_conn_id="postgres_conn",
overwrite=True,
)
demo_dag = demo_with_s3_and_csv()

Was this page helpful?