Hi All, I was wondering is someone could help me identify why some of my tasks are pending. I have the following flow running on prefect cloud with a dask backend. for some reason, the task scheduler has not started 2/4 of the tasks. thanks for any advice

Hi @Vincent, what RunConfig and Executor are you using?

KubernetesRun with a DaskExecutor

Did you configure the resources? This seems like the resources are limited. This thread may give ideas.

Indeed I did set the resource limits for the scheduler to be pretty beefy.

I’d have to ask some team members who know more about k8s. I’ll respond to you tomorrow.

Hey @Vincent -- are they hanging indefinitely? Are you using timeouts? Are you returning any values from the functions that may not be serializable or use a lot of memory? Is your

DaskExecutor

being created with any arguments?

The tasks will usually proceed if there is some stimulus. ie. another task finishes. However, this is undesirable in the case where 1 task can hold up the entire flow. I do not specify any special args other than the cluster_class and adaptive_kwargs. The return of each function is a small and serializable, (ie. a file path).

Hmm. What are your cluster class/adaptation args?