Hi everyone, I have a very intriguing question - I...
# prefect-community
Hi everyone, I have a very intriguing question - I have registered my flow on a k8s prefect-server and it's using Dask Gateway as the executor. The flow is actually working just fine, to my surprise... However, I can't see the Dask cluster being created by Dask Gateway in the GCP UI. Therefore, there are no dask workers to execute the job, yet the job is still running fine? Is it even using the Dask executor I specified when I registered the flow?
The flows are still running
But no dask workers are created to handle the job
Hi! Please don’t directly ping engineers, we have a support rotation. Can you provide the code you’ve used to configure your flow run/execution?
Got it - thanks. My bad. I believe I showed this snippet to Kyle before, and he said it looked good and to notify the thread if there are any problems. Code snippet:
Copy code
auth = BasicAuth(password=secrets['dask_gateway_secret'])
    gateway = Gateway(
    options = gateway.cluster_options()
    options.image = args.dask_gateway_docker_image

    cluster = gateway.new_cluster(options)

    for flow in flows:
        flow.storage = storage
        flow.executor = DaskExecutor(
            client_kwargs={'security': cluster.security}
        path = storage.add_flow(flow)
        print(f'storing flow {flow.name} at {path} at the image.')

    storage = storage.build()

    for flow in flows:
        flow.register(project_name=args.project, build=False)
It looks like you’re defining your cluster at flow registration time instead of flow runtime. Take a look at https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-temporary-cluster
upvote 1
I'll give that a try - thanks!
Got it working - thanks for your assistance @Zanie. I used a custom function that creates the cluster via Dask Gateway and passed it into the
argument. That's very cool functionality! Thanks