Hey guys, @Dana Merrick and I are using

LocalDaskExecutor

in combination with

RunNamespacedJob

to launch a bunch of trivially parallelizable child processes on Kubernetes. However, it looks like only 2 child jobs are getting run at a time. Is there a way to increase the parallelism // are we doing this right? 😄

Hey @Gabe Grand ! How do you define the LocalDaskExecutor? Did you specify the number of workers?

nope just defaults

LocalDaskExecutor()

Ah ok I think we just need to up the number of workers from the default. Will look how.

oh nice! well that’s easy 🙂

Of course!

just thinking down the line a bit here - is there a way to specify different

num_workers

for different tasks? we want high

num_workers

for these

RunNamespacedJob

tasks, but there are other tasks in the flow that I could see breaking with too many threads

i think you just override the

executor

in the flow definition

That would be cool if that worked @Dana Merrick. Will ask around and see if there’s any other ways and get back to you on this one.

@Gabe Grand check out https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html#task-concurrency-limiting

@Kevin Kho @Dylan Could you sanity check our design pattern? We’re running

t1 = RunNamespacedJob.map(body=[b1, b2, …])

to create a bunch of parallel K8s jobs, and then we have some downstream tasks that depend on

t1

. Not sure if this is the best way to achieve MapReduce-style parallelism with Prefect.

So workers are allocated at initialization and overriding in the flow will not work because the latest one is the one saved for execution. Task concurrency limiting is the only way to achieve that.

@Dylan ah right! I remember you mentioned this was one of the main use cases for tags. thanks!

Yes I think the design pattern is correct. The immediate non-mapped task after t1 will perform the reduce.

We’re currently using a custom setup for Prefect server, but will think about Cloud to take advantage of the global concurrency limit tagging feature