within prefect. Is it guaranteed that the order is preserved across multiple maps, like a python
multiprocessing.pool.map
? Or is it not a guarantee that the order is preserved, like
multiprocessing.pool.imap_unordered
. I couldn't find whether this was guaranteed or not in the
prefect docs
.
k
Kevin Kho
08/20/2021, 6:41 PM
Hey @Kyle Hansen, is it not guaranteed. There is no priority ordering on the map.
k
Kyle Hansen
08/20/2021, 6:45 PM
So just to make sure, if I do a
func.map([a, b, c])
the result I get back might be
[result_b, result_c, result_a]
?
I know they might not finish computing in the same order, but from my small scale tests, it seems I'm getting back
[result_a, result_b, result_c]
which I'm guessing is just an odd artifact of being lucky?
k
Kevin Kho
08/20/2021, 6:46 PM
Oh that order of results in the list is guaranteed but the order of execution is not. So if you chain mapped tasks
a = A.map()
and
B.map(a)
, the output and input will connect as expected
👍 1
k
Kyle Hansen
08/20/2021, 6:50 PM
Ah ok. Thanks for confirming that. This is kind of what I thought from my testing, but I wanted to make sure that was guaranteed. And I'm assuming that order is also preserved with a flat-mapping?
So if I have
section of the documentation and subsequent examples (eg. Flat-mapping), but I couldn't find anywhere it explicitly says that is guaranteed to be true.
k
Kevin Kho
08/20/2021, 6:53 PM
That one i’m not 100% sure. I’ll ask the team for you
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.