Tom Klein
07/08/2022, 9:34 AMLocalDaskExecutor
for our flow, but to be able to limit the parallelisation (due to each task requiring a lot of resources when being run) to - for example - only two tasks at a time, is that possible? I read the docs but still donโt fully understand if we have to use a DaskExecutor
for this, and if so - would it by default run locally (if we donโt give it any other config)?
and kind of tangenial (and not directly related to prefect) but is there some advantage of something like the AWS fargate cluster for dask of a k8s dask cluster? is the former just easier to set up or something?Pierre-Edouard
07/08/2022, 9:37 AMflow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
Tom Klein
07/08/2022, 9:42 AMPierre-Edouard
07/08/2022, 9:46 AMAnna Geller
07/08/2022, 11:18 AMTom Klein
07/08/2022, 11:48 AMAnna Geller
07/08/2022, 11:53 AM