I am trying to understand the differences between the TaskRunner and the CloudTaskRunner . It seems that these two Runners schedule tasks a slightly differently. ie. I have a flow that runs in parallel Locally, but fails to run in parallel when using the DaskKubernetesEnvironment. Is there a summary of the differences in scheduling ?
c
Chris White
10/02/2020, 6:28 PM
Hi Vincent - task runners perform no scheduling; it sounds like your dask kubernetes environment has only 1 worker possibly? Can you check the number of workers, threads and processes you have configured for your cluster?
v
Vincent
10/02/2020, 7:00 PM
I have this set up with the DaskKubernetesEnvironment where I set min_workers =2 and max_workers=10 (and the rest as default settings). There are parts of my DAG that execute serially in cloud while they run in parallel locally (with threads). I know that the downstream tasks definitely take advantage of the cluster because I see multiple tasks running.
c
Chris White
10/02/2020, 7:03 PM
Can you confirm that you have work stealing turned on in your cluster? What version of Prefect are you running with?
v
Vincent
10/02/2020, 7:09 PM
Is there a setting beyond the
DaskKubernetesEnvironment(work_stealing=True)
? I am using 13.8
c
Chris White
10/02/2020, 7:11 PM
Nope that looks correct to me
v
Vincent
10/02/2020, 10:36 PM
I think I resolved this issue. It seems that when I map a flattened list of lists, the computation works via threads instead of via multiple workers.