save_file
AstroExport a database table as a CSV or Parquet file to local storage, Amazon S3, or Google Cloud Storage.
Access Instructions
Install the Astro provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
output_file_pathRequiredstrThe target file path.
tableastro.sql.table.TableThe ``Table`` to use as a source for offloading to a file.
output_conn_idstrThe ID of the configured Airflow Connection to use to save the output file (if S3 of GCS).
output_file_formatstrThe file format to use when writing the output file. Valid values are 'csv' or 'parquet'.
overwriteboolFlag whether or not to overwrite the output file if it already exists.
task_idstrThe ID of the task.
Documentation
Export a database table as a CSV or Parquet file to local storage, Amazon S3, or Google Cloud Storage.
- Example:
- @dag(default_args=default_args,schedule_interval=None,start_date=datetime(2021, 1, 1),tags=["demo"],)def demo_with_s3_and_csv():t1 = load_file(path="s3://tmp9/homes.csv",file_conn_id="my_aws_conn",database="astro",output_table_name="expected_table_from_s3",)t2 = load_file(path="tests/data/homes.csv",database="astro",output_table_name="expected_table_from_csv",)save_file(output_file_path="s3://tmp9/homes.csv",table=t1,input_conn_id="postgres_conn",overwrite=True,)save_file(output_file_path="tests/data/homes_output.csv",table=t2,input_conn_id="postgres_conn",overwrite=True,)demo_dag = demo_with_s3_and_csv()