Hey folks, apologies if this is a rookie question, as I’m just getting started with Prefect. What I’m aiming to do is set up a flow where each task is a container run on a separate google cloud compute instance (where the instance is spun up on task start and destroyed on task end, instance type & config depends on the task). It seems like this lies somewhere between the purpose of the Docker agent and the Kubernetes agent, where the former runs task as containers but only locally, while the latter can create new instances but requires a full Kubernetes setup (which we currently don’t use). Am I understanding this correctly? Is the workflow I described similar to what someone has built with Prefect?

That’s what I’m working to set up currently; we’ve opted for Kubernetes due to the ease of implementing auto-scaling; our flows run early in the day, after which we can make the hardware scale down etc.

Cheers for the response @Wilson Bilkovich! That article is super interesting. I think our use-case at the moment is even simpler than what you’re describing. We know the resource requirements of a given task upfront, so all we really need to do is wrap our Python task with tasks to create and destroy the necessary GCE instance. Something along the lines of airflow’s ComputeEngineStartInstanceOperator + the ability to pul an image and run the container on said instance. I guess it wouldn’t be too hard to write a new task class that does this though

Yeah, it seems like a thing the Task Library could have in it, but just happens to not have yet

We’d welcome PR’s for the task library 🙂