Why are our flow limited to a concurrency of 2 tasks? We are using prefect cloud with a Kubernetes Agent running 2 workers. For some reason it only runs 2 tasks concurrently. I have tried setting the executor to DaskExecutor() and LocalDaskExecutor(). The behaviour stays the same The tasks I am running in this example are just waiting for 10 seconds, so resource constraints can’t really be the issue. I assume it’s limiting it to 1 concurrent task per Kubernetes worker node, but haven’t been able to confirm that. Anyone know any possible reasons for this?

Hey @Hilary Roberts, could I see how you set up the executor and run configuration?

Sure, what do you need to see exactly? I defined the flow like this:

with Flow(
        "name",
        schedule,
        run_config=KubernetesRun(),
        executor=DaskExecutor()
) as flow:
   ...

And also with the LocalDaskExecutor

Did you try bumping up the number of workers like:

flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)

I'll give that a go

Or

flow.executor = DaskExecutor(
    cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}
)

Nice! The Local one worked, thanks! I'll try the second option too.

The

LocalDaskExecutor

does not use

distributed

and is a Local Dask multiprocessing pool while the

DaskExecutor

works on

distributed

so you would use it if you had multiple machines (a cluster), which I think you do in your setup.

Got it, so I guess the local just runs it on the master node.

Yep!