Vaibhav Ariyur

01/05/2022, 4:26 PM
Hi all, in Prefect is the following kind of dynamic task generation possible: user passes in the following parameter: [0,1,[2,3],0,[1,2,3]] and in doing so dynamically creates the following flow: task0 --> task1 --> parallel(task2,task3) --> task0 --> parallel(task1,task2,task3) I know dynamic mapping can take the user input and dynamically create tasks that are in parallel, but is this possible too?

Kevin Kho

01/05/2022, 4:32 PM
I don’t think this is possible in current Prefect because of the DAG requirement. Dynamic task generation like this is one of the motivations behind Orion (Prefect 2.0). For now, the best thing you can do is Task Looping and Mapping , but this specific case will be a bit tricky. You need to break that input into chunks and then use looping or mapping. You also kind of need to know the structure beforehand