b

    Benjamin Filippi

    1 year ago
    Hi guys, there a seemingly simple thing I can’t manage to do...... I want to cascade mappings: I have two lists: L1=[A,B] L2=[1,2,3] I want to generate 6 tasks, expected execution plan is as follow: Execute (A,1) (A,2) (A,3) (B,1)(B,2)(B,3) Reduce A from first 3 reduce B from last 3 Reduce (A,B) What simple trick am I missing?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Hi @Benjamin Filippi, There's a few ways you could do this. • If the lists
    L1
    and
    L2
    are static (fully known at flow build time), you could loop over them and manually construct the tasks without using mapped tasks at all. • If they're not static (result from some task known only at runtime), you might have a task that takes in the lists and generates a list of pairs
    (A, 1), (A, 2)...
    , then map a task over those pairs. Your reducer function would then need to filter out tasks to apply the reductions only to
    A
    tasks, only to
    B
    tasks, etc... If the lists are static and small, I'd probably go with the first option as the code would look more like what you'd write in a simple python script. The latter would work with prefect mapping natively if you need more dynamic tasks.
    b

    Benjamin Filippi

    1 year ago
    Yes this is the road I am taking, using list(itertools.product(l1,l2)) And then writing a custom reduce Thanks for your answer