b

    Bob Colner

    2 years ago
    question about mapping tasks -is it possible for a task to be 'mapped' over multiple iterators? (think task with 2 parameters and I want to 'map' over both with a list of param values)
    j

    josh

    2 years ago
    🤔 Is something like this what you’re after?
    from prefect import task, Flow, Parameter
    
    @task
    def do_something(a, b):
        print(a)
        print(b)
    
    with Flow("two-map") as flow:
        a = Parameter('a')
        b = Parameter('b')
    
        do_something.set_upstream(a, key="a", mapped=True)
        do_something.set_upstream(b, key="b", mapped=True)
    
    flow.run(parameters={"a": [1, 2, 3], "b": [4, 5, 6]})
    b

    Bob Colner

    2 years ago
    humm, i'm not useing prefect params (don't need to set them at runtime)
    with Flow(name='backfill-flow', result=result_store) as flow:
    
        backfill_task_result = backfill_task.map(
            symbol=['SPY','GLUU','IHI','NVDA'],
            backfill_date=['2020-07-01','2020-07-02'],
            tick_type=unmapped('trades')
        )
    just trying to map over 2 separate lists that map unto 2 separate (normal) parameters
    when I run the above flow it maps over the
    backfill_date
    param but not the
    symbol
    param. Hence my question
    Jeremiah

    Jeremiah

    2 years ago
    Hi @Bob Colner - Prefect’s map has a
    zip
    semantic for multiple arguments, rather than a
    product
    semantic (see this related issue). To achieve your goal, I would recommend precomputing the combinations of
    symbol
    and
    backfill_dates
    you require, and mapping over that result. For example
    backfill_task.map(
      list(itertools.product(
        ['SPY', 'GLUU', 'IHI', 'NVDA'], 
        ['2020-07-01', '2020-07-02']
      )
    )
    You could also perform that product step in a Prefect task to do it dynamically (I think the linked issue has an example of that)
    b

    Bob Colner

    2 years ago
    thanks, this makes sense
    Jeremiah

    Jeremiah

    2 years ago
    ok great!
    b

    Bob Colner

    2 years ago
    I do think it would be a nice to have native support for multiple mapped params, and (perhaps) for nested mappings
    Jeremiah

    Jeremiah

    2 years ago
    The product semantic implicitly requires gathering all data, since it can’t be done in parallel (in Spark it would require a
    shuffle
    ), so I think we will always recommend using a separate task like the one in the linked issue. Otherwise it would unexpectedly break map’s parallel semantic by secretly necessitating a “reduce” step
    However I do actually have a draft PR open for nested mapping: https://github.com/PrefectHQ/prefect/pull/2898