question about mapping tasks -is it possible for a task to be 'mapped' over multiple iterators? (think task with 2 parameters and I want to 'map' over both with a list of param values)

🤔 Is something like this what you’re after?

from prefect import task, Flow, Parameter

@task
def do_something(a, b):
    print(a)
    print(b)

with Flow("two-map") as flow:
    a = Parameter('a')
    b = Parameter('b')

    do_something.set_upstream(a, key="a", mapped=True)
    do_something.set_upstream(b, key="b", mapped=True)

flow.run(parameters={"a": [1, 2, 3], "b": [4, 5, 6]})

humm, i'm not useing prefect params (don't need to set them at runtime)

Hi @Bob Colner - Prefect’s map has a

zip

semantic for multiple arguments, rather than a

product

semantic (see this related issue). To achieve your goal, I would recommend precomputing the combinations of

symbol

and

backfill_dates

you require, and mapping over that result. For example

backfill_task.map(
  list(itertools.product(
    ['SPY', 'GLUU', 'IHI', 'NVDA'], 
    ['2020-07-01', '2020-07-02']
  )
)

thanks, this makes sense

ok great!

I do think it would be a nice to have native support for multiple mapped params, and (perhaps) for nested mappings

The product semantic implicitly requires gathering all data, since it can’t be done in parallel (in Spark it would require a

shuffle

), so I think we will always recommend using a separate task like the one in the linked issue. Otherwise it would unexpectedly break map’s parallel semantic by secretly necessitating a “reduce” step