Benjamin Filippi
10/06/2020, 3:53 PMJim Crist-Harif
10/06/2020, 5:29 PML1
and L2
are static (fully known at flow build time), you could loop over them and manually construct the tasks without using mapped tasks at all.
• If they're not static (result from some task known only at runtime), you might have a task that takes in the lists and generates a list of pairs (A, 1), (A, 2)...
, then map a task over those pairs. Your reducer function would then need to filter out tasks to apply the reductions only to A
tasks, only to B
tasks, etc...
If the lists are static and small, I'd probably go with the first option as the code would look more like what you'd write in a simple python script. The latter would work with prefect mapping natively if you need more dynamic tasks.Benjamin Filippi
10/06/2020, 5:32 PM