This time, I wrote a blog post on how to manage de...
# show-and-tell
a
This time, I wrote a blog post on how to manage dependencies between data pipelines. I mainly use here
FlowRunTask
to trigger single flows in order. This helped my team to organize the flows between several layers of jobs in our data warehouse: staging area, business logic, data mart. I also compare how this can be done in Airflow vs. in Prefect and Prefect version proved to be much simpler. I hope this may be useful to some of you. Greetings from Berlin! 🙂 https://towardsdatascience.com/managing-dependencies-between-data-pipelines-in-apache-airflow-prefect-f4eba65886df
❤️ 4
😍 2
👏 12
👋 6
🚀 14
h
@Anna Geller (old account) Hi, loved the blog post! simple and insightful, it helped me a lot. I wanted to ask if you know how to get (within the UI) from a specific run of a child flow to its parent? In other words, how can you know that a specific run of your ‘staging_area’ flow was triggered by the ‘MasterFlow’ and not by itself or by another flow? Thanks!
a
I would look at the timestamp in the logs. Afaik, there is no drill down to go from parent logs to child flow logs in the UI, so I guess you need to look at the timestamp and drill down manually. So far for me, this wasn't too difficult, because I was only looking at it when something failed and when it failed, you will see it in the UI by red bars 🙂
h
Thanks, I kind of wish there was someway to do it 😅