https://prefect.io logo
Title
a

Anna Geller (old account)

09/04/2020, 8:10 PM
This time, I wrote a blog post on how to manage dependencies between data pipelines. I mainly use here
FlowRunTask
to trigger single flows in order. This helped my team to organize the flows between several layers of jobs in our data warehouse: staging area, business logic, data mart. I also compare how this can be done in Airflow vs. in Prefect and Prefect version proved to be much simpler. I hope this may be useful to some of you. Greetings from Berlin! 🙂 https://towardsdatascience.com/managing-dependencies-between-data-pipelines-in-apache-airflow-prefect-f4eba65886df
❤️ 4
😍 2
🚀 14
👏 12
👋 6
h

Hagai Arad

09/29/2020, 12:35 PM
@Anna Geller (old account) Hi, loved the blog post! simple and insightful, it helped me a lot. I wanted to ask if you know how to get (within the UI) from a specific run of a child flow to its parent? In other words, how can you know that a specific run of your ‘staging_area’ flow was triggered by the ‘MasterFlow’ and not by itself or by another flow? Thanks!
a

Anna Geller (old account)

09/29/2020, 12:56 PM
I would look at the timestamp in the logs. Afaik, there is no drill down to go from parent logs to child flow logs in the UI, so I guess you need to look at the timestamp and drill down manually. So far for me, this wasn't too difficult, because I was only looking at it when something failed and when it failed, you will see it in the UI by red bars 🙂
h

Hagai Arad

09/29/2020, 1:20 PM
Thanks, I kind of wish there was someway to do it 😅