Hi, I'm using the

DaskExecutor

with

dask-kubernetes

to spin up a dask cluster when a flow starts and running it to process 5,000 files via a mapped task with a few final aggregation tasks. I sometimes see (via the Prefect UI) that the dask cluster appears to spin down near the end of the task, but before it's completely done leaving a few tasks stuck pending with no workers to process them. Has anyone seen this before? If so, how did you solve it?

Hi @Adam Lewis! This only happens sometimes but works other times? Is there anything in the logs that would give more details? Is it the aggregation tasks that get stuck in pending or the mapped tasks?

Thanks for pointing to that thread. That could be a similar problem since things were stopped near the end of the large mapped run. Things I could try, 1) upping the resources of the kubernetes job associated with this flow (or at least look at its memory usage to see how much memory it's using as the mapped tasks are gathered) 2) exposing the dask dashboard and watching it near the end of the mapped tasks 3) trying changing from using threads to processes.

Yes those are all the thoughts I have for now. Decreasing the resources may also do something.