An Hoang
04/13/2020, 4:15 PMdf_B
is obtained from df_A
by passing it through some function A_to_B_func()
. df_D
is made by merging df_B
with df_C
etc.
If it’s a complex function, yes we can write prefect Tasks to visualize it, but usually it’s an aggregate of small pandas functions and merge, so it would be a pain to wrap each of those into its own prefect task. Maybe I would need to write a complex context manager/decorator or a meta class wrapper around dataframe to record these operations automatically into a prefect task graph? Sorry the idea is not so clear in my head right now…Alex Cano
04/13/2020, 4:22 PMAn Hoang
04/13/2020, 5:13 PMLeo Meyerovich (Graphistry)
04/13/2020, 5:23 PMAn Hoang
04/13/2020, 7:15 PM