Bob Colner
07/03/2020, 4:13 PMjosh
07/03/2020, 4:17 PMfrom prefect import task, Flow, Parameter
@task
def do_something(a, b):
print(a)
print(b)
with Flow("two-map") as flow:
a = Parameter('a')
b = Parameter('b')
do_something.set_upstream(a, key="a", mapped=True)
do_something.set_upstream(b, key="b", mapped=True)
flow.run(parameters={"a": [1, 2, 3], "b": [4, 5, 6]})
Bob Colner
07/03/2020, 4:22 PMwith Flow(name='backfill-flow', result=result_store) as flow:
backfill_task_result = backfill_task.map(
symbol=['SPY','GLUU','IHI','NVDA'],
backfill_date=['2020-07-01','2020-07-02'],
tick_type=unmapped('trades')
)
backfill_date
param but not the symbol
param. Hence my questionJeremiah
07/03/2020, 4:38 PMzip
semantic for multiple arguments, rather than a product
semantic (see this related issue). To achieve your goal, I would recommend precomputing the combinations of symbol
and backfill_dates
you require, and mapping over that result. For example
backfill_task.map(
list(itertools.product(
['SPY', 'GLUU', 'IHI', 'NVDA'],
['2020-07-01', '2020-07-02']
)
)
Bob Colner
07/03/2020, 4:47 PMJeremiah
07/03/2020, 4:56 PMBob Colner
07/03/2020, 4:57 PMJeremiah
07/03/2020, 4:59 PMshuffle
), so I think we will always recommend using a separate task like the one in the linked issue. Otherwise it would unexpectedly break map’s parallel semantic by secretly necessitating a “reduce” step