Benjamin Filippi10/06/2020, 3:53 PM
Jim Crist-Harif10/06/2020, 5:29 PM
are static (fully known at flow build time), you could loop over them and manually construct the tasks without using mapped tasks at all. • If they're not static (result from some task known only at runtime), you might have a task that takes in the lists and generates a list of pairs
, then map a task over those pairs. Your reducer function would then need to filter out tasks to apply the reductions only to
(A, 1), (A, 2)...
tasks, only to
tasks, etc... If the lists are static and small, I'd probably go with the first option as the code would look more like what you'd write in a simple python script. The latter would work with prefect mapping natively if you need more dynamic tasks.
Benjamin Filippi10/06/2020, 5:32 PM