Thanks for link, but in our case we have several tasks, which depends on each other, and we want looping this group of tasks while result of first task in group is true. Example : first task get list of files from kafka , second task run as .map over dask cluster. Equivalent is while(len(files)) > 0 do staff.
z
Zach
11/12/2020, 4:31 PM
There might be an easier way, but you can run a flow from a flow. So at the end of your tasks, you can check if you want to "loop", and if you do you can call the same flow again with the new parameters you want.
https://docs.prefect.io/core/idioms/flow-to-flow.html
This does slow things down a bit though, since it takes time to schedule a prefect flow, so if you can loop within your flow without scheduling a new flow that would be much better.
b
bral
11/12/2020, 4:38 PM
thanks! we just want to avoid starting the entire flow due to re-registration in order to reduce overhead costs. but we don't know how :)
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.