Hans Lellelid
11/03/2023, 6:07 PMDaskTaskRunner
(just a default dask cluster, not connecting to existing cluster). (We're calling <task>.submit()
and then using fut.wait()
in a loop to ensure it is complete.) What we're seeing is that we have a zombie python processes under prefect-agent pid ticking up every minute -- seems to be correlated to this. Is there something we should be doing additionally to gracefully close down the Dask workers so that we don't end up with zombie processes? After a few days of this, it ends up crashing the server .... while I think we have a workaround (there's no need to use Dask for this; threads would work fine), I'd love to understand what might be going on as we do want to use Dask for other longer-running prefect flows.Hans Lellelid
11/03/2023, 6:08 PMHans Lellelid
11/09/2023, 1:03 AM