https://prefect.io logo
Title
a

Avi A

06/08/2020, 9:08 PM
Another question! I notice that some of the tasks on my flow are running multiple times, even though they have finished successfully (so they finish successfully 5 times). What could cause such a behavior?
c

Chris White

06/08/2020, 9:10 PM
This is almost always caused by Dask rescheduling work; we already have a “version locking” feature in Prefect Cloud that we hope to make more widely available in the very near future that prevents your tasks from running twice will still preserving a clean user experience
a

Avi A

06/08/2020, 9:13 PM
thanks, you mention it runs twice but I see it’s running 5 times. have you seen such a behavior too?
c

Chris White

06/08/2020, 9:13 PM
are you using the
LocalDaskExecutor
by any chance?
a

Avi A

06/08/2020, 9:30 PM
yes I am
but is it really different than setting a local “cluster” and using
DaskExecutor
?
c

Chris White

06/08/2020, 9:34 PM
yea that executor is known to rerun tasks, but I’ll admit I’ve never heard of 5 reruns! Yea it’s a total different scheduler
a

Avi A

06/08/2020, 9:35 PM
so do you recommend I stop being lazy and move to
DaskExecutor
?
c

Chris White

06/08/2020, 9:39 PM
Yea I think that is the simplest solution to your rerun issue
:prefect: 1
🚀 1
a

Avi A

06/21/2020, 11:08 AM
@Chris White following up here. I am using
DaskExecutor
now. The scheduler and the worker themselves are still local, but I specify them in different processes in the system
I’m still seeing successful tasks running several times for no reason.
moreover, it seems it doesn’t schedule that many tasks concurrently (the scheduler has 4*8 threads available), and the flow hangs after running a few tasks (there are 3*250 mapped tasks to run, and it only ran 65 of them, and hasn’t scheduled the rest yet)