https://prefect.io logo
Title
m

Michael

08/30/2021, 8:14 AM
Hey folks, apologies if this is a rookie question, as I’m just getting started with Prefect. What I’m aiming to do is set up a flow where each task is a container run on a separate google cloud compute instance (where the instance is spun up on task start and destroyed on task end, instance type & config depends on the task). It seems like this lies somewhere between the purpose of the Docker agent and the Kubernetes agent, where the former runs task as containers but only locally, while the latter can create new instances but requires a full Kubernetes setup (which we currently don’t use). Am I understanding this correctly? Is the workflow I described similar to what someone has built with Prefect?
👀 1
w

Wilson Bilkovich

08/30/2021, 1:19 PM
That’s what I’m working to set up currently; we’ve opted for Kubernetes due to the ease of implementing auto-scaling; our flows run early in the day, after which we can make the hardware scale down etc.
But if you want to do it just with the Google docker registry + their auto-scaling features, it should work.
Here’s somebody’s blog post about using K8S to do it; I don’t think it would be too painful to adapt it to ‘bare’ Docker, you just need to translate the node-group stuff into different terrms https://medium.com/google-cloud/scale-your-kubernetes-cluster-to-almost-zero-with-gke-autoscaler-9c78051cbf40
Another approach would be to write another subclass over in Dask, like
KubeCluster
and
HelmCluster
but configured to use Google instances instead of pods; that would be super clean but may be more work.
m

Michael

08/30/2021, 3:01 PM
Cheers for the response @Wilson Bilkovich! That article is super interesting. I think our use-case at the moment is even simpler than what you’re describing. We know the resource requirements of a given task upfront, so all we really need to do is wrap our Python task with tasks to create and destroy the necessary GCE instance. Something along the lines of airflow’s ComputeEngineStartInstanceOperator + the ability to pul an image and run the container on said instance. I guess it wouldn’t be too hard to write a new task class that does this though
w

Wilson Bilkovich

08/30/2021, 3:02 PM
Yeah, it seems like a thing the Task Library could have in it, but just happens to not have yet
k

Kevin Kho

08/30/2021, 3:20 PM
We’d welcome PR’s for the task library 🙂
👍 1