Ron Van Buskirk
11/20/2019, 3:43 PM* having separate tasks for building each table (450+ tasks!):
Check_table1 -> ifelse -> Build_table1 |-> Check_table2 -> ifelse -> Build_table2 ...
|-> Check_table3 -> ifelse -> Build_table3 ...
|-> Check_table4 -> ifelse -> Build_table4 ...
* subclassing the Postgres execute task to create a check-and-build task:
Check_and_build_table1 |-> Check_and_build_table2 ...
|-> Check_and_build_table3 ...
|-> Check_and_build_table4 ...
* having a small number of tasks (check timestamp and existence, conditional, build) and using the map function to iterate the building of the required tables:
Check_table1 -> ifelse -> Build_table1 |-> Check_table.map(x) -> ifelse -> Build_table(x) ...
...
...
Still really new to Prefect... are any of these any good? Are there any other best practices I should consider?Zachary Hughes
11/20/2019, 4:03 PMRon Van Buskirk
11/20/2019, 6:33 PM