Emma Rizzi

    Emma Rizzi

    9 months ago
    Hi! I'm trying to create a custom VM image that launches a prefect agent on spawn, is there any records of such work already done here ? I haven't found anything but still worth to ask My goal is to simulate an elastic cluster with prefect: have a flow that launches a instance with this image, start a new sub-flow with this new agent, and deprovision the instance Almost everystep is validated, I just need to build the VM image in
    raw
    format and I got lost here, if anyone has experience with it I'd gladly take some advice!
    Amanda Wee

    Amanda Wee

    9 months ago
    Out of curiosity: why do you want to do this? If the reason is that the sub-flow requires a large amount of resources that you want to provision then deprovision as fast as possible, then it seems to me that using say, ECS agent to run flows in short-lived Fargate tasks might achieve the same thing in a more common pattern.
    Anna Geller

    Anna Geller

    9 months ago
    @Emma Rizzi can you leave this VM running? I think spinning up agents on demand will cause you trouble, Prefect assumes that agents are long-running processes which are always on 24/7. But what you need seems to be: a way distribute work across those machines, correct?
    Emma Rizzi

    Emma Rizzi

    9 months ago
    @Amanda Wee @Anna Geller I'm working on the dias Wekeo, this cloud environment does not provide elastic clusters so I try to simulate it throught Prefect as I can access to the Morpheus Api to deploy instances. My team needs to deploy data processing pipelines on wekeo, and we can't use another provider as a requirement for this project (mainly for satellite data contained on this cloud). I'm trying to integrate prefect as I had some good experiences with it previously it seems doable, and to allow to integrate multiple cloud providers in future projects In order to reduce resource consumption and cost, i would have one instance constantly running (the master agent) which would launch the others instances when needed
    i know it might seem weird to use prefect this way, but for context if I can make this work I think it would be a really interesting improvement for the earth observation community to integrate multiple dias in a single tool! most of the dias developped today offer really basic cloud computing options 🙂
    Amanda Wee

    Amanda Wee

    9 months ago
    It sounds like you need a Wekeo agent for Prefect! 😄 After all, it'll be more efficient if the master agent could launch instances to run the flows, rather than launch instances to run the secondary agents to run the flows.
    Anna Geller

    Anna Geller

    9 months ago
    @Emma Rizzi Prefect does not orchestrate resources though. It orchestrate your flows, not the underlying compute. This is what things like Kubernetes, Mesos, and various solutions from cloud providers are for. I think the best solution in your use case would be to build a Kubernetes cluster that would leverage your Wekeo VMs. You could leverage all the Kubernetes concepts to ensure that your cluster scales as needed (by adding or removing nodes) and a single
    KubernetesAgent
    could be used to ensure that your flow runs are deployed as Kubernetes jobs to the cluster. While Prefect would manage the states of your flow runs, Kubernetes would manage the underlying compute resources based on your cpu and memory requests you specify on your
    KubernetesRun
    . Having said that, I’m aware that deploying such cluster wouldn’t be trivial if you have never done this, but this would be a much cleaner and sustainable approach.
    Emma Rizzi

    Emma Rizzi

    9 months ago
    @Amanda Wee gosh i didn't even consider this option 😮 i was really stubborn about launching a docker agent dynamically Thank you and @Anna Geller for the suggestions, I'm not sure if the Kubernetes option is doable but it does sound like the best one!