Jeff Brainerd
11/06/2020, 2:08 AMLocalEnvironment
with a DaskExecutor
. We are managing a semi-persistent dask cluster ourselves (i.e. the same cluster runs multiple flow runs). We would like to shift to a scheme that creates an ephemeral dask cluster per flow run. Is the current best practice to use LocalEnvironment
with
DaskExecutor(
cluster_class="dask_cloudprovider.FargateCluster", ...
)
?
Should I stay away from DaskCloudProviderEnvironment
?
Thanks for any info! 🙏Dylan
Jeff Brainerd
11/06/2020, 3:45 PMDylan