Hi all,
I’m here to migrate from airflow to prefect because of the latency.
I have some questions here.
I have 2 layers for the process.
1. data processing
2. modeling
I have many data processed every minute and also have many models processed every minute.
A model have dependencies to some data.
i.e)
data_A - model_A
data_B -
data_A - model_B
data_C -
I tried to make each data processing into a flow -> N flows
and also each modeling process into a flow -> M flows
Then I had a parent flow setting dependencies between data flows and model flows.
However this had long latency (I suspect poll_interval)
So I made a huge flow which contains all data processing and modeling process.
this made the flow much faster but hard to monitor with the UI.
Do you think there are some way to achieve this goal?