Hi everyone, I'm trying to setup a Fargate Dask client following the example from prefect in here with Dark Cloud Provider https://docs.prefect.io/orchestration/execution/dask_cloud_provider_environment.html#process. The problem is since the scheduler uses by default ports 8786 and 8787 for security reasons these ports are blocked at the AWS account I'm using, and because of that is causing the scheduler to time out. Is there any way to change these ports? Couldn't find anything in the docs just for Azure integration. Thanks!

You can forward additional keyword arguments to the underlying dask cluster class by passing them to

DaskCloudProviderEnvironment

. See the dask docs for options: https://cloudprovider.dask.org/en/latest/api.html. It looks like you probably want to set

scheduler_port

and

dashboard_port

.

Thanks for the reply Jim, although I don't see scheduler_port option for cluster_kwargs only for

AzureMLCluster

could you provide any example?

Ah, I missed that this wasn't exposed for

FargateCluster

. Would you mind opening an issue in the dask-cloudprovider repo to standardize this? https://github.com/dask/dask-cloudprovider/issues In the meantime, I believe configuring this via

scheduler_extra_args

. Something like:

executor = DaskExecutor(
    cluster_class="dask_cloudprovider.FargateCluster",
    cluster_kwargs={
        "scheduler_extra_args": ["--port", "1234", "--dashboard-address", ":5678"]
    }
)

Sure I'll try that thank you