Hey community,
We have a mapped task in our flow, spawning several thousand mapped tasks, whose output it then forwarded to one task (unmapped). We noticed that these tasks are being called twice. Is this expected in some way? This happened to me before, but I thought this was already solved.
prefect 13.1
running on dask-kubernetes
m
Mariia Kerimova
12/09/2020, 3:21 PM
Hello @Avi A! I would suggest to upgrade to the latest Prefect version, it should solve the issue. If it will not work, it would be great if you open an issue on the repo with reproducible example.
a
Avi A
12/09/2020, 9:42 PM
Thanks. Turns out I was wrong, we were using version 13.18. we will upgrade to 13.19, but are you aware of anything that was fixed that might have caused this kind of issue ?
m
Mariia Kerimova
12/09/2020, 10:04 PM
The potential for duplicated runs could be related to expired idempotency keys, it was fixed about a month ago.
Also I would suggest to double check how many agents are running. If you are using multiple agents, differentiate them by labels.
a
Avi A
12/09/2020, 10:22 PM
Thanks! We will take a look
cc @Vladimir Zoubritsky
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.