Chris
07/26/2019, 7:10 AMthere appear to be 1 leaked semaphores to clean up at shutdown
. Also, rather than killing the flow, or restarting the failed task, it restarts upstream tasks which were successfully run. Does anyone know how what the cause of failure might be and whether it’s possible to only restart the failed task?Jeremiah
07/26/2019, 1:44 PMsemaphore
warning directly, but the behavior — in which a Dask worker’s failure prompts Dask to rerun upstream functions — is one that we now have a replicable example for and are considering raising an issue in the Distributed repo. It appears that this is a brief moment of assumed idempotency in the Distributed scheduler — all other Dask retry logic can be disabled via config, but not this one. To be clear, I don’t think this is a Dask bug, per se, but I do think it needs a way to turn off the behavior.Chris
07/26/2019, 1:51 PMJeremiah
07/26/2019, 1:52 PMChris
07/26/2019, 1:55 PM