Fábio Alves
08/04/2020, 4:07 PMJim Crist-Harif
08/04/2020, 4:12 PMDaskCloudProviderEnvironment
. See the dask docs for options: https://cloudprovider.dask.org/en/latest/api.html. It looks like you probably want to set scheduler_port
and dashboard_port
.DaskExecutor
instead. This should already work, but we haven't added deprecation warnings yet.
In your case it might look like:
executor = DaskExecutor(
cluster_class="dask_cloudprovider.FargateCluster",
cluster_kwargs={...} # extra parameters
)
Fábio Alves
08/04/2020, 4:30 PMAzureMLCluster
could you provide any example?flow.run(executor=DaskExecutor(cluster.scheduler.address, cluster.cluster_kwargs={}),
parameters={"x": list(range(10))})
Jim Crist-Harif
08/04/2020, 4:38 PMFargateCluster
. Would you mind opening an issue in the dask-cloudprovider repo to standardize this? https://github.com/dask/dask-cloudprovider/issues
In the meantime, I believe configuring this via scheduler_extra_args
. Something like:
executor = DaskExecutor(
cluster_class="dask_cloudprovider.FargateCluster",
cluster_kwargs={
"scheduler_extra_args": ["--port", "1234", "--dashboard-address", ":5678"]
}
)
Fábio Alves
08/04/2020, 4:49 PM