Hello - I am mostly deployed and things are working well - running prefect server with DaskExecutor. I have created a custom wrapper around the Flow class so that I can easily set the run environment for local development vs production flows. I have a function that is creating the dask scheduler/worker through dask gateway - all works, EXCEPT that the dask cluster is also spun up on registration. Is there an easy way to prevent this? I am wondering if I have to make a custom dask class and use that as part of the executor, rather than a custom function that creates it as part of initiating the flow.

If it is spun up during registration, are you calling flow.run during registration?

Just to add: this documentation page may be helpful - it shows how to pass various options depending on your Dask setup https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-temporary-cluster

Ah - this seems like the key piece maybe

This can be either a string specifying the import path to the cluster class (e.g. "dask_cloudprovider.aws.FargateCluster"), the cluster class itself, or a function for creating a custom cluster

Maybe you can try this GatewayCluster

flow.executor = DaskExecutor(
    cluster_class=GatewayCluster(),
    cluster_kwargs={"n_workers": 4, "image": "my-prefect-image"},
)

yeah I have one example showing this with Fargate:

import prefect
from dask_cloudprovider.aws import FargateCluster # or ECSClusterpre

def fargate_cluster(n_workers=4):
    """Start a fargate cluster using the same image as the flow run"""
    return FargateCluster(n_workers=n_workers, image=prefect.context.image)

flow.executor = DaskExecutor(
    cluster_class=fargate_cluster,
    cluster_kwargs={"n_workers": 4}
)

got it, so I think it’s probably going to be like

dask_gateway.GatewayCluster

or something in this case (psuedo-code)

Yep that’s right.

Confirmed that just passing

dask_gateway.GatewayCluster

with proper params works - I don’t think there is a way to set the number of workers without scaling it, but I was able to set the adapt pretty easily.

Yeah I’m not seeing it either looking at the docs. I think you’re right