https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Amogh Kulkarni

07/06/2022, 6:21 PM
Hi Team. In prefect 2.0, is there a way to visualize the flows like how we could do in prefect 1.0 with flow.visualize()?
1
k

Kevin Kho

07/06/2022, 6:21 PM
Not really because there is no DAG requirement so there is only a DAG after the flow run
z

Zanie

07/06/2022, 6:26 PM
(We’re interested in still providing this in the future)
a

Amogh Kulkarni

07/06/2022, 6:28 PM
Thanks Kevin and Michael. Good to know that it will be provided in the future.
a

Anna Geller

07/06/2022, 6:39 PM
@Amogh Kulkarni can you say what would you like to see in this graph/what's your use case? is it for data engineering use case to see which data gets moved between systems (say extract data from A, load data to B, trigger job C) or is it really purely for workflow lineage, i.e. show function doing transformation A, then transformation B, then transformation C, etc?
a

Amogh Kulkarni

07/06/2022, 11:50 PM
It’s mainly to track lineage. The task dependency in our flows is very complex. We usually use it to check our task dependency before we run it on our local machine and before we register the flow to prod EKS cluster.
a

Anna Geller

07/07/2022, 9:14 PM
thanks a lot @Amogh Kulkarni - what do you mean when you say lineage? do you mean dependencies between datasets, tables in a data warehouse and/or ML models? or do you mean dependencies between tasks e.g. to track the state of data transformations? or something entirely different?
a

Amogh Kulkarni

07/08/2022, 7:54 PM
@Anna Geller I mainly meant workflow lineage. ie. dependencies between tasks. We have complex flows. Image one flow with 20 tasks. One task depends on 5 parallel/concurrent tasks to complete. After that there are a series of concurrent and sequential tasks. Some of them pass dataframes to their dowstream tasks. Using flow.visualize(), I first get a graph diagram and see if I have set the dependencies correctly and the right dataframes are passed. Once I have confirmed this I then run the flow on my local machine. I generate the diagram once again before registering the flow on prod.
I couldn’t do this in airflow I was glad that prefect 1.0 offered it.
a

Anna Geller

07/08/2022, 8:26 PM
thanks so much for giving us more context about that 🙌
👍 1
9 Views