Thread
#prefect-community
    a

    Avi A

    2 years ago
    Another question! I notice that some of the tasks on my flow are running multiple times, even though they have finished successfully (so they finish successfully 5 times). What could cause such a behavior?
    Chris White

    Chris White

    2 years ago
    This is almost always caused by Dask rescheduling work; we already have a “version locking” feature in Prefect Cloud that we hope to make more widely available in the very near future that prevents your tasks from running twice will still preserving a clean user experience
    a

    Avi A

    2 years ago
    thanks, you mention it runs twice but I see it’s running 5 times. have you seen such a behavior too?
    Chris White

    Chris White

    2 years ago
    are you using the
    LocalDaskExecutor
    by any chance?
    a

    Avi A

    2 years ago
    yes I am
    but is it really different than setting a local “cluster” and using
    DaskExecutor
    ?
    Chris White

    Chris White

    2 years ago
    yea that executor is known to rerun tasks, but I’ll admit I’ve never heard of 5 reruns! Yea it’s a total different scheduler
    a

    Avi A

    2 years ago
    so do you recommend I stop being lazy and move to
    DaskExecutor
    ?
    Chris White

    Chris White

    2 years ago
    Yea I think that is the simplest solution to your rerun issue
    a

    Avi A

    2 years ago
    @Chris White following up here. I am using
    DaskExecutor
    now. The scheduler and the worker themselves are still local, but I specify them in different processes in the system
    I’m still seeing successful tasks running several times for no reason.
    moreover, it seems it doesn’t schedule that many tasks concurrently (the scheduler has 48 threads available), and the flow hangs after running a few tasks (there are 3250 mapped tasks to run, and it only ran 65 of them, and hasn’t scheduled the rest yet)