Hello 🙋 If we wanna use a

LocalDaskExecutor

for our flow, but to be able to limit the parallelisation (due to each task requiring a lot of resources when being run) to - for example - only two tasks at a time, is that possible? I read the docs but still don’t fully understand if we have to use a

DaskExecutor

for this, and if so - would it by default run locally (if we don’t give it any other config)? and kind of tangenial (and not directly related to prefect) but is there some advantage of something like the AWS fargate cluster for dask of a k8s dask cluster? is the former just easier to set up or something?

I don't know if it's what you are searching but you can define the number of thread for localDaskExecutor:

flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)

@Pierre-Edouard are you sure? the docs seem to suggest otherwise

To be honest I just read it faslty last week here: https://docs.prefect.io/orchestration/flow_config/executors.html#localdaskexecutor

the suggestion to limit the number of workers is good you could also leverage concurrency limits to solve that https://docs.prefect.io/orchestration/flow-runs/concurrency-limits.html#task-run-limits

@Anna Geller in that case i suggest that the docs are updated to reflect this extra option, no? anyway, i think tags etc. could be useful but seems like kind of an indirect way to control this when i just want it to apply for this specific flow (it's more of a property of the specific tasks being run, rather than a property of an external resource being interacted with) i guess in general we should probably be executing it with a Dask cluster to begin with, but we're just not there yet

I see what you mean. There are trade-offs here as always - concurrency limits are more generic, while executor settings are more to control resources rather than concurrency