append
AstroAppend the results of a source table onto a target table. This function assumes there are no conflicts between the schemas of both tables.
Access Instructions
Install the Astro provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
append_tableRequiredastro.sql.table.TableThe Table containing the rows to append to a target table.
main_tableRequiredastro.sql.table.TableThe target ``Table`` to which rows should be appended.
columnsList[str]A list of columns to use for the append operation. If no columns are specified, all columns from the append_table will be used.
casted_columnsdictA mapping of column:datatype pairs which will be used to cast columns as particular datatypes when appending i.e. ``CAST(column AS datatype)``.
Documentation
Having transformed a table, you might want to append the results to a reporting table. An example of this might be to aggregate daily data on a "main" table that analysts use for timeseries analysis. The append function merges tables assuming that there are no conflicts. You can choose to merge the data 'as-is' or cast it to a new value if needed. Note that this query will fail if there is a merge conflict.
- Example:
- foo = append(conn_id="postgres_conn",database="postgres",append_table=APPEND_TABLE,columns=["Bedrooms", "Bathrooms"],casted_columns={"Age": "INTEGER"},main_table=MAIN_TABLE,)