09/14/2021, 10:28 AM
Hi all, I’m here to migrate from airflow to prefect because of the latency. I have some questions here. I have 2 layers for the process. 1. data processing 2. modeling I have many data processed every minute and also have many models processed every minute. A model have dependencies to some data. i.e) data_A - model_A data_B - data_A - model_B data_C - I tried to make each data processing into a flow -> N flows and also each modeling process into a flow -> M flows Then I had a parent flow setting dependencies between data flows and model flows. However this had long latency (I suspect poll_interval) So I made a huge flow which contains all data processing and modeling process. this made the flow much faster but hard to monitor with the UI. Do you think there are some way to achieve this goal?
👋 6