Hello all, any chance someone can offer advice or reference examples on spinning up a Kubernetes deployment on Google Cloud Platform as a preliminary step to deploying ephemeral DaskExecutor flows? My org has a limited budget and we’re trying to minimize cloud costs by scaling our infrastructure to zero when we’re not running large web scraping jobs once or twice a week. I’ve looked at Dask Cloud Provider, which has GCP support and could work. But we also wanted to explore a k8s solution before settling on a final strategy. I realize this question is not directly related to Prefect, but hoping someone might have experience with this kind of approach (perhaps using something like Terraform?). Any advice is greatly appreciated!

Hi @Serdar Tumgoren, I don’t know the native GCP approach but I’ve used Coiled to do this because the Coiled cluster has

shutdown_on_close

, which terminates it after the Flow.

executor = DaskExecutor(
    cluster_class=coiled.Cluster,
    cluster_kwargs={
        "software": "kvnkho/prefect",
        "shutdown_on_close": True,
        "name": "prefect-cluster",
    },
)
flow.executor = executor
flow.register(...)

@Kevin Kho Thanks Kevin! Yep, the Dask CloudProvider looks like it could be a good solution, but one of our requirements is the ability to mount a Google Cloud storage volume (as opposed to mounting buckets). From my initial exploration of Cloud Provider, that seemed like it would require a fair bit of customization, whereas I think I read somewhere that k8s has “simpler” (all things relative) support for mounting storage volumes. Coiled looks very interesting so I’ll check that out as well. Ultimately, we just want the ability to spin up some a certain number of machines, mount a storage volume and access SQL/NoSQL datastores during a flow run — and then have those cloud resources spun down in the simplest way possible. Coiled looks like it could get us there, so I’ll explore options for mounting storage volumes using that tool as well. We could also find ways to work around the requirement of mounting storage, in which case Dask Cloud Provider sounds like it might be the easiest option. Let me know if you have any thoughts on the above, and thanks for the reply and helpful suggestions!

Seems like questions for the Coiled folks. Wanna talk to them? They’re in this channel. I use them in my demos for Prefect + Dask and really think it’s the easiest way to go. I just don’t know about the mount.

@Kevin Kho That’d be great! Is there a certain person we can ping?

Hello all. Multi tasking a bit, but trying to take a look.

@Greg Smith Many thanks! I’ll also start reviewing Coiled docs to get a better feel for the library and platform. Thanks!

Hi @Serdar Tumgoren. Great to meet you. I think Coiled can do most of what you want, but I am going to have to look into mounting Google Cloud storage volumes. In principle, if you can do it with Python or Dask, you should be able to do it on coiled. And, we can let you run either in our GCP account (passing through compute costs) or (still beta) using your own account. That seems likley to be best, in terms of getting to your storage. If you're OK with this, I'll DM you an invitation to the Coiled Community slack, and then I can follow up with you there via DM (on Monday if that's ok) with more information on your specific question. meanwhile, you can 'read the docs'. 🙂 We try to make those useful, but it's also good to learn if there are questions we don't yet answer. Give me a moment for that Slack invite.

@Greg Smith That all sounds great! The storage volume issue is a nice-to-have on our end. We can work around it by using datastore on GCP, but if you do know of a solution that also supports mounting storage volumes that’d be amazing. I’ll start leveling up on Coiled today and will join the Coiled slack as well once you send the invite. Then we can chat early next week at a time that works for ya. Many thanks for the speedy response!