I am trying to understand the differences between the TaskRunner and the CloudTaskRunner . It seems that these two Runners schedule tasks a slightly differently. ie. I have a flow that runs in parallel Locally, but fails to run in parallel when using the DaskKubernetesEnvironment. Is there a summary of the differences in scheduling ?

Hi Vincent - task runners perform no scheduling; it sounds like your dask kubernetes environment has only 1 worker possibly? Can you check the number of workers, threads and processes you have configured for your cluster?

I have this set up with the DaskKubernetesEnvironment where I set min_workers =2 and max_workers=10 (and the rest as default settings). There are parts of my DAG that execute serially in cloud while they run in parallel locally (with threads). I know that the downstream tasks definitely take advantage of the cluster because I see multiple tasks running.

Can you confirm that you have work stealing turned on in your cluster? What version of Prefect are you running with?

Is there a setting beyond the

DaskKubernetesEnvironment(work_stealing=True)

? I am using 13.8

Nope that looks correct to me

I think I resolved this issue. It seems that when I map a flattened list of lists, the computation works via threads instead of via multiple workers.