Hey guys, I'm using Prefect with Dask, using the built-in DaskExecutor. I see in the Dask gui that sometimes tasks 'linger'. They don't get cleaned up. Over time, this eats memory. Restarting the Dask Scheduler helps, but ideally I would see the tasks getting cleaned up by the system itself. Is this a known issue and is there a way to keep the memory clean? We're using dask==2.9.0 and prefect==0.7.3

Hi @Jan Harkema! I don’t think we’ve heard of this issue yet — you might try upgrading to prefect 0.9.3 just to see if that helps but we haven’t knowingly made any changes that would effect this. Let us know if you gain any more insight or if we can help out in any way!

Thanks for your reply! I'll schedule an upgrade to 0.9.3 after the weekend and keep investigating. If I know more, I will give an update.

I'm also having an issue with running out of memory on the DaskExecutor. I have 5 mapped tasks that are looping concurrently, just making an http request each, every 20 seconds, and eventually, this happens:

Heartbeat failed to start.  This could result in a zombie run.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 58, in inner
    p = subprocess.Popen(self.heartbeat_cmd, env=current_env)
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1482, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

executing in Fargate on a container with 2G mem Prefect 0.9.4