Hi Team, I need to apply these Dask Config settings to my DaskExecutor before it gets created. I’m using Prefect Server. Both Prefect and Dask are setup on a GKE cluster. Can this be done using the

cluster_kwargs

in DaskExecutor: https://github.com/PrefectHQ/prefect/blob/6b59d989dec33aad8c62ea2476fee519c32f5c63/src/prefect/executors/dask.py#L88-L89 https://docs.prefect.io/api/latest/executors.html#daskexecutor

These can be set with env variables right? You can add the env variables to the DaskExecutor.

Yes they can be set with env variables. How to add those to DaskExecutor? An example would really help.

I think this syntax works. The cluster_kwargs is for things that the Dask Cluster class like this:

executor = DaskExecutor(
        cluster_class="dask_cloudprovider.FargateCluster",
        cluster_kwargs={
            "image": "prefecthq/prefect:latest",
            "n_workers": 5,
            ...
        },
    )

so those can passed to

FargateCluster

.

I’m creating a cluster myself using the Dask Gateway object and it points to the Gateway server setup on GKE.

from dask_gateway import BasicAuth, Gateway
    
def create_cluster(min_workers, max_workers, docker_image, proxy_address):
        # connect to dask-gateway proxy
        auth = BasicAuth(password=secrets['dask_gateway_secret'])
        gateway = Gateway(address=proxy_address, auth=auth)

        options = gateway.cluster_options()
        options.image = docker_image
        options.worker_memory = 16.0  # max is 16.0
        options.worker_cores = 5  # max is 8

        cluster = gateway.new_cluster(options)
        cluster.adapt(minimum=min_workers, maximum=max_workers)
        return cluster

This

cluster

is passed to the

cluster_class

param in

DaskExecutor

This seems fine. Does this not work?

This works but I haven’t passed any Dask Config’s from here into this yet. Ex: `distributed.comm.timeouts.tcp=30s`_ _