Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi! I'm trying to create a custom VM image that launches a prefect agent on spawn, is there any records of such work already done here ? I haven't found anything but still worth to ask
My goal is to simulate an elastic cluster with prefect: have a flow that launches a instance with this image, start a new sub-flow with this new agent, and deprovision the instance
Almost everystep is validated, I just need to build the VM image in `raw`  format and I got lost here, if anyone has experience with it I'd gladly take some advice!

Out of curiosity: why do you want to do this? If the reason is that the sub-flow requires a large amount of resources that you want to provision then deprovision as fast as possible, then it seems to me that using say, ECS agent to run flows in short-lived Fargate tasks might achieve the same thing in a more common pattern.

<@U02NW8KPFBK> can you leave this VM running? I think spinning up agents on demand will cause you trouble, Prefect assumes that agents are long-running processes which are always on 24/7. But what you need seems to be: a way distribute work across those machines, correct?

<@U01DJDK6AQ7> <@U02H1A95XDW> I'm working on the dias Wekeo, this cloud environment does not provide elastic clusters so I try to simulate it throught Prefect as I can access to the Morpheus Api to deploy instances. My team needs to deploy data processing pipelines on wekeo, and we can't use another provider as a requirement for this project (mainly for satellite data contained on this cloud). I'm trying to integrate prefect as I had some good experiences with it previously it seems doable, and to allow to integrate multiple cloud providers in future projects
In order to reduce resource consumption and cost, i would have one instance constantly running (the master agent) which would launch the others instances when needed

i know it might seem weird to use prefect this way, but for context if I can make this work I think it would be a really interesting improvement for the earth observation community to integrate multiple dias in a single tool! most of the dias developped today offer really basic cloud computing options :slightly_smiling_face:

It sounds like you need a Wekeo agent for Prefect! :smile: After all, it'll be more efficient if the master agent could launch instances to run the flows, rather than launch instances to run the secondary agents to run the flows.

<@U02NW8KPFBK> Prefect does not orchestrate resources though. It orchestrate your flows, not the underlying compute. This is what things like Kubernetes, Mesos, and various solutions from cloud providers are for.

I think the best solution in your use case would be to build a Kubernetes cluster that would leverage your Wekeo VMs. You could leverage all the Kubernetes concepts to ensure that your cluster scales as needed (by adding or removing nodes) and a single `KubernetesAgent` could be used to ensure that your flow runs are deployed as Kubernetes jobs to the cluster. While Prefect would manage the states of your flow runs, Kubernetes would manage the underlying compute resources based on your cpu and memory requests you specify on your `KubernetesRun`.

Having said that, I’m aware that deploying such cluster wouldn’t be trivial if you have never done this, but this would be a much cleaner and sustainable approach.

<@U01DJDK6AQ7> gosh i didn't even consider this option :open_mouth: i was really stubborn about launching a docker agent dynamically
Thank you and <@U02H1A95XDW> for the suggestions, I'm not sure if the Kubernetes option is doable but it does sound like the best one!