Hi everyone, I have a very intriguing question - I have registered my flow on a k8s prefect-server and it's using Dask Gateway as the executor. The flow is actually working just fine, to my surprise... However, I can't see the Dask cluster being created by Dask Gateway in the GCP UI. Therefore, there are no dask workers to execute the job, yet the job is still running fine? Is it even using the Dask executor I specified when I registered the flow?

Hi! Please don’t directly ping engineers, we have a support rotation. Can you provide the code you’ve used to configure your flow run/execution?

Got it - thanks. My bad. I believe I showed this snippet to Kyle before, and he said it looked good and to notify the thread if there are any problems. Code snippet:

auth = BasicAuth(password=secrets['dask_gateway_secret'])
    gateway = Gateway(
        address=args.dask_proxy_address,
        auth=auth
    )
    options = gateway.cluster_options()
    options.image = args.dask_gateway_docker_image

    cluster = gateway.new_cluster(options)
    cluster.scale(5)

    for flow in flows:
        flow.storage = storage
        flow.executor = DaskExecutor(
            address=cluster.scheduler_address,
            client_kwargs={'security': cluster.security}
        )
        path = storage.add_flow(flow)
        print(f'storing flow {flow.name} at {path} at the image.')

    storage = storage.build()

    for flow in flows:
        flow.register(project_name=args.project, build=False)

It looks like you’re defining your cluster at flow registration time instead of flow runtime. Take a look at https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-temporary-cluster

I'll give that a try - thanks!

Wonderful!