https://prefect.io logo
Title
v

Vadym Dytyniak

11/30/2022, 2:46 PM
Hi. Is it correct statement that Prefect 2 map function does not guarantee the order? Example:
l1 = task1.map([1, 2, 3])
l2 = task2.map(l1)

task3.map(l1=l1, l2=l2)
Prefect does not guarantee that in
task3
list
l1
elements will be passed with corresponding
l2
elements?
a

Anna Geller

11/30/2022, 3:23 PM
it does; if you see any issue with ordering here, it would be great if you open an issue with MRE
v

Vadym Dytyniak

11/30/2022, 4:29 PM
For ConcurrentTaskRunner as well?
a

Anna Geller

11/30/2022, 4:45 PM
concurrent will try to run tasks concurrently but the submission is in the same order you provided and the results are ordered so e.g. numbers as results from extract will be passed in the right order to transform
from prefect import flow
from prefect_dask import DaskTaskRunner
from flows.parallel.tasks import extract, transform, load


@flow(task_runner=DaskTaskRunner())
def dask_flow_map():
    numbers = extract.submit()
    transformed_numbers = transform.map(numbers)
    load.submit(numbers=transformed_numbers)
regardless of which task was submitted first etc
but you need either .map() or .submit() to submit tasks to the task runner
otherwise things run in the flow run process
v

Vadym Dytyniak

11/30/2022, 4:49 PM
Hmm, interesting, can't create MRE(not reproducible), but see issues in prod flow
a

Anna Geller

11/30/2022, 4:52 PM
gotcha, hard to fix if we can't have MRE, if you could dig deeper and file a bug report if you find something more specific, we will def look into it 🙌
🙌 1