This is an open discussion and more of a suggestio...
# data-tricks-and-tips
s
This is an open discussion and more of a suggestion-seeking question. What is the best way to handle Schema Drift when the data type of a column is changed in the source while loading the data in the destination table? Here’s the context, the work process is ETL using an orchestration tool(Prefect) where the load is done in Snowflake. Here are my ideas: • compare the precedence value of the data type in the destination column and source column whichever is of lower precedence, modify the column to that data type. For example:
col_a
in destination holds
integer
data type and
col_a
in source has float. Since float data type has a lower precedence value than an integer, modify the integer to float. • create a new column with a suffixed, For example:
col_a
in destination holds integer data type and
col_a
in source has float, create a new column with
col_a_float
. Later map the source
col_a
to
col_a_float
👍 1