r

    Riley Hun

    1 year ago
    Hi everyone, I have a very intriguing question - I have registered my flow on a k8s prefect-server and it's using Dask Gateway as the executor. The flow is actually working just fine, to my surprise... However, I can't see the Dask cluster being created by Dask Gateway in the GCP UI. Therefore, there are no dask workers to execute the job, yet the job is still running fine? Is it even using the Dask executor I specified when I registered the flow?
    The flows are still running
    But no dask workers are created to handle the job
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! Please don’t directly ping engineers, we have a support rotation. Can you provide the code you’ve used to configure your flow run/execution?
    r

    Riley Hun

    1 year ago
    Got it - thanks. My bad. I believe I showed this snippet to Kyle before, and he said it looked good and to notify the thread if there are any problems. Code snippet:
    auth = BasicAuth(password=secrets['dask_gateway_secret'])
        gateway = Gateway(
            address=args.dask_proxy_address,
            auth=auth
        )
        options = gateway.cluster_options()
        options.image = args.dask_gateway_docker_image
    
        cluster = gateway.new_cluster(options)
        cluster.scale(5)
    
        for flow in flows:
            flow.storage = storage
            flow.executor = DaskExecutor(
                address=cluster.scheduler_address,
                client_kwargs={'security': cluster.security}
            )
            path = storage.add_flow(flow)
            print(f'storing flow {flow.name} at {path} at the image.')
    
        storage = storage.build()
    
        for flow in flows:
            flow.register(project_name=args.project, build=False)
    Michael Adkins

    Michael Adkins

    1 year ago
    It looks like you’re defining your cluster at flow registration time instead of flow runtime. Take a look at https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-temporary-cluster
    r

    Riley Hun

    1 year ago
    I'll give that a try - thanks!
    Got it working - thanks for your assistance @Michael Adkins. I used a custom function that creates the cluster via Dask Gateway and passed it into the
    cluster_class
    argument. That's very cool functionality! Thanks
    Michael Adkins

    Michael Adkins

    1 year ago
    Wonderful!