https://prefect.io logo
#prefect-community
Title
# prefect-community
t

Tom Klein

07/08/2022, 9:34 AM
Hello ๐Ÿ™‹ If we wanna use a
LocalDaskExecutor
for our flow, but to be able to limit the parallelisation (due to each task requiring a lot of resources when being run) to - for example - only two tasks at a time, is that possible? I read the docs but still donโ€™t fully understand if we have to use a
DaskExecutor
for this, and if so - would it by default run locally (if we donโ€™t give it any other config)? and kind of tangenial (and not directly related to prefect) but is there some advantage of something like the AWS fargate cluster for dask of a k8s dask cluster? is the former just easier to set up or something?
p

Pierre-Edouard

07/08/2022, 9:37 AM
I don't know if it's what you are searching but you can define the number of thread for localDaskExecutor:
Copy code
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
๐Ÿ˜ฎ 1
๐Ÿ™Œ 1
t

Tom Klein

07/08/2022, 9:42 AM
@Pierre-Edouard are you sure? the docs seem to suggest otherwise
is it just undocumented?
it does seem like in another example it is being referrenced: https://docs.prefect.io/orchestration/flow_config/executors.html#localdaskexecutor
p

Pierre-Edouard

07/08/2022, 9:46 AM
Sorry, I send the same link ๐Ÿ˜‰ .
๐Ÿ‘ 1
a

Anna Geller

07/08/2022, 11:18 AM
the suggestion to limit the number of workers is good you could also leverage concurrency limits to solve that https://docs.prefect.io/orchestration/flow-runs/concurrency-limits.html#task-run-limits
t

Tom Klein

07/08/2022, 11:48 AM
@Anna Geller in that case i suggest that the docs are updated to reflect this extra option, no? anyway, i think tags etc. could be useful but seems like kind of an indirect way to control this when i just want it to apply for this specific flow (it's more of a property of the specific tasks being run, rather than a property of an external resource being interacted with) i guess in general we should probably be executing it with a Dask cluster to begin with, but we're just not there yet
๐Ÿ™Œ 1
๐Ÿ‘ 1
a

Anna Geller

07/08/2022, 11:53 AM
I see what you mean. There are trade-offs here as always - concurrency limits are more generic, while executor settings are more to control resources rather than concurrency
โœ… 1
4 Views