Hey, I’ve noticed that unused allocated memory isn’t freed after a task/flow is run, so if I run a scheduled flow which contains a memory-intensive task, that memory is constantly allocated from the first time the flow runs. I’ve found some workarounds (manually deleting variables at the end of a task’s run function), but this doesn’t always work (e.g. if the output of one task is passed as input to another). Is there a better workaround? I also have a similar issue when running large-scale flows with the Dask executor. It seems like memory is not freed between tasks - I found this relevant issue https://github.com/dask/dask/issues/3247 which suggests that the pool used by a Dask worker to complete a task is not closed after each task. This causes issues with large-scale flows as even if I split the data into small chunks and use these as mapped args to a task, the leaked memory accumulates with each task run and ends up causing workers to die. Has anyone experienced anything similar?

How many tasks does your Flow have and how large of data are they returning? I experienced something similar once but realized my tasks were returning data of size ~4MB and I had ~12,000 of them so that wasn’t feasible. I ended up finding a workaround and my Flow ran just fine

Thanks for the reply, the tasks don’t return any data (rather, each task writes a chunk of data to disk), and there are ~1000 of them (running with a cluster of 60 workers).

Has there been any update on this? I’m having the same problem

I haven’t had time to dig into this on our side, but for what it’s worth I doubt this is a prefect specific issue; is there any chance this is a memory leak in a dependency you’re using ( dask array / numpy) and are you on the most up-to-date version of dask? Anecdotally, I’ve run some very large flows on dask quite recently with no issue