https://prefect.io logo
Title
b

Bob Colner

07/06/2020, 6:40 PM
Hi all, I'm struggling to get the 'zip' mapping pattern to work, I'm probably missing something basic here. I'm tying to map a task with 2 parameters. I'm looking at the reply to this github issue (https://github.com/PrefectHQ/prefect/issues/1986) where it is suggested to use the
itertools.product
function to map over multiple tasks. However, I can't get this to work in a simple example:
import itertools
from prefect import Flow, task
from prefect.engine.executors import LocalDaskExecutor


@task(checkpoint=False)
def cross_product(x, y) -> list:
    return list(itertools.product(x, y))


@task(checkpoint=False)
def concat(a:str, b:str) -> str:
    return a + b


a = ['d', 'o', 'g']
b = ['c', 'a', 't']

with Flow(name='zip-map-test') as flow:
    
    result = concat.map(cross_product(a, b))


executor = LocalDaskExecutor(scheduler='threads')

flow_state = flow.run(executor)
when I run this flow I get:
TypeError: missing a required argument: 'b'
k

Kyle Moon-Wright

07/06/2020, 6:48 PM
For that particular error, the
concat
function you're mapping is looking for two arguments - whereas you only have
cross_product(a, b)
as the first arg.
b

Bob Colner

07/06/2020, 6:50 PM
yeah, my understanding is that the the cross_prod task returns a list of tuples:
[('d', 'c'),
 ('d', 'a'),
 ('d', 't'),
 ('o', 'c'),
 ('o', 'a'),
 ('o', 't'),
 ('g', 'c'),
 ('g', 'a'),
 ('g', 't')]
so for each task there are 2 strings available -in normal python I would use the 'splat' * to unpack the tuple
I guess I assumed the the prefect map know how to do this (the github issue does not use any *splat)
@Dylan do you have any ideas? trying to implement the suggestion you gave my last week. thanks!
d

Dylan

07/06/2020, 7:01 PM
@Bob Colner In the example above, you might need to adjust your flow to be:
with Flow(name='zip-map-test') as flow:
    cross_result = cross_product(a,b)
    result = concat.map(cross_result)
and move the unpacking of the tuple into the
concat
task
Prefect defers computation to execution time, as opposed to definition time
b

Bob Colner

07/06/2020, 7:18 PM
ok, cool. let me try this. FIY -this will be nice to add to the mapping docs
that worked. thanks