Hi guys, there a seemingly simple thing I can’t ma...
# prefect-community
Hi guys, there a seemingly simple thing I can’t manage to do...... I want to cascade mappings: I have two lists: L1=[A,B] L2=[1,2,3] I want to generate 6 tasks, expected execution plan is as follow: Execute (A,1) (A,2) (A,3) (B,1)(B,2)(B,3) Reduce A from first 3 reduce B from last 3 Reduce (A,B) What simple trick am I missing?
Hi @Benjamin Filippi, There's a few ways you could do this. • If the lists
are static (fully known at flow build time), you could loop over them and manually construct the tasks without using mapped tasks at all. • If they're not static (result from some task known only at runtime), you might have a task that takes in the lists and generates a list of pairs
(A, 1), (A, 2)...
, then map a task over those pairs. Your reducer function would then need to filter out tasks to apply the reductions only to
tasks, only to
tasks, etc... If the lists are static and small, I'd probably go with the first option as the code would look more like what you'd write in a simple python script. The latter would work with prefect mapping natively if you need more dynamic tasks.
Yes this is the road I am taking, using list(itertools.product(l1,l2)) And then writing a custom reduce Thanks for your answer