question about mapping tasks -is it possible for a...
# prefect-community
b
question about mapping tasks -is it possible for a task to be 'mapped' over multiple iterators? (think task with 2 parameters and I want to 'map' over both with a list of param values)
j
🤔 Is something like this what you’re after?
Copy code
from prefect import task, Flow, Parameter

@task
def do_something(a, b):
    print(a)
    print(b)

with Flow("two-map") as flow:
    a = Parameter('a')
    b = Parameter('b')

    do_something.set_upstream(a, key="a", mapped=True)
    do_something.set_upstream(b, key="b", mapped=True)

flow.run(parameters={"a": [1, 2, 3], "b": [4, 5, 6]})
👍 1
b
humm, i'm not useing prefect params (don't need to set them at runtime)
Copy code
with Flow(name='backfill-flow', result=result_store) as flow:

    backfill_task_result = backfill_task.map(
        symbol=['SPY','GLUU','IHI','NVDA'],
        backfill_date=['2020-07-01','2020-07-02'],
        tick_type=unmapped('trades')
    )
just trying to map over 2 separate lists that map unto 2 separate (normal) parameters
when I run the above flow it maps over the
backfill_date
param but not the
symbol
param. Hence my question
j
Hi @Bob Colner - Prefect’s map has a
zip
semantic for multiple arguments, rather than a
product
semantic (see this related issue). To achieve your goal, I would recommend precomputing the combinations of
symbol
and
backfill_dates
you require, and mapping over that result. For example
Copy code
backfill_task.map(
  list(itertools.product(
    ['SPY', 'GLUU', 'IHI', 'NVDA'], 
    ['2020-07-01', '2020-07-02']
  )
)
👍 1
You could also perform that product step in a Prefect task to do it dynamically (I think the linked issue has an example of that)
b
thanks, this makes sense
j
ok great!
b
I do think it would be a nice to have native support for multiple mapped params, and (perhaps) for nested mappings
j
The product semantic implicitly requires gathering all data, since it can’t be done in parallel (in Spark it would require a
shuffle
), so I think we will always recommend using a separate task like the one in the linked issue. Otherwise it would unexpectedly break map’s parallel semantic by secretly necessitating a “reduce” step
👍 1
However I do actually have a draft PR open for nested mapping: https://github.com/PrefectHQ/prefect/pull/2898