Hi guys,
I am looking at the possibility to have loops in a flow.
The existing LOOP feature that I found in the doc is interesting, but would it be possible to do the same with several tasks ?
Like for example Task A produces a result, then Task B uses this result to produce another one, then Task C uses this second result, does its own thing with it and, if a condition is not met, calls A again.
I know it sort of contradicts the meaning of a DAG, but it would be really useful for us... Is it an existing feature?
k
Kevin Kho
06/25/2021, 4:35 PM
Hi @thebuleon29, it feels like it would be a lot simpler if you placed Task B and Task C inside the LOOP with Task A in this case. Does your use case prohibit this?
t
thebuleon29
06/28/2021, 9:20 AM
Sort of. We are reviewing workflow engines for a project, and one of the customer's requirements is to be able to do it.
I managed to do it in Airflow using sub dags, so I am quite surprised to see Prefect not having an equivalent mechanism.
k
Kevin Kho
06/28/2021, 3:51 PM
Could you explain the sort of a bit more? Just making sure you know that tasks can be used in other tasks by calling the
task.run()
method.
Kevin Kho
06/28/2021, 6:51 PM
Btw, I asked the team and one thing you can do here is have taskC start a new flow if the condition is not met. This would keep your tasks A, B, C as independent.
t
thebuleon29
07/01/2021, 3:48 PM
Hi,
Thanks, starting a new flow was what I was looking for
k
Kevin Kho
07/01/2021, 4:22 PM
This can be done through the Client where you can use the
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.