Hello, we often find ourselves having to explicitly reschedule flows even when we have enough workers on our DaskKubernetes cluster. is there a common reason this happens?

Hey Diego, do you have an error message?

It’s not an error message. the flow just refuses to continue, with tasks in Pending despite there being dask workers available

Are you using a mapped tasks and how many elements does it have?

No mapped tasks. It’s a 200 or so node graph

This commonly happens then with out of memory issues. Have you checked the pod?

which pod?

Sorry, the Dask scheduler pod specifically (I assume it would die before the workers)

the dask scheduler pod is on all the time and i can send jobs to it manually, so the problem seems to be on the prefect side

i think we are making changes to the prefect code for the DaskExecutor cuz there are stuff that could be more efficient, but most of the changes tend to be around mapping where repeated work is being done.