Mike Marinaccio
11/06/2020, 3:33 PMECR Task
and run_config
recently added, which sound like a potential solution.
Thanks for the input!Joe Schmid
11/06/2020, 4:12 PMon_execute()
callback. See this example in the docs: https://docs.prefect.io/orchestration/execution/dask_cloud_provider_environment.html#advanced-example-dynamic-worker-sizing-from-parameters-tls-encryption
IMPORTANT NOTE: I believe the Prefect team is moving away from this environment and instead recommending use of a LocalEnvironment
with a DaskExecutor
, e.g.
DaskExecutor(
cluster_class="dask_cloudprovider.FargateCluster", ...
)
I haven't had a chance yet to look at implementing the same "dynamically set Dask worker sizing based on Flow parameters" idea using that approach.Mike Marinaccio
11/06/2020, 4:25 PM