Hello! I'm considering to use Prefect to orchestrate VM as I work on a Cloud called WEkEO Dias and does not offer dynamic provisionning of instances. This typical workflow would be executed on a master VM : 1. start a VM 2. launch a flow run on this VM 3. shutdown the VM I thought about using the function

create_flow_run

to launch part 2 synchronized with steps 1 and 3. Considering I can configure some script to deploy the Agent on VM start, is it possible ? I'm concerned about the main flow run starting before the VM's agent exists If you have suggestions on better ways to implement this with Prefect I'm interested ! So far i only used basic ECS agent 🙂

@Emma Rizzi is it OK for you if the VM is running 24/7? The agent must be always on because it needs to poll Prefect API for new flow runs every 10 seconds. If you want a more pay-as-you-go provisioning model, you could try ECS again, or the new Google Vertex agent - Vertex could be potentially a good option for you since afaik it spins up entire VMs for the flows. However, it would run on GCP rather than on the “WEkEO Dias” 🙂 Docs on Vertex: • https://docs.prefect.io/orchestration/agents/vertex.html • https://docs.prefect.io/api/latest/run_configs.html#vertexrun

@Anna Geller thanks for your answer. At the moment it is mandatory to use Wekeo because of user requirements. In this workflow the master VM would run 24/7, but not the worker VM. If step 1 is over only after the agent is running on the worker VM, step 2 would be called when agent is ready. I never tried the function

create_flow_run

, I thought this would work unless this function performs checks too early in the workflow (before the second agent is up) ?

I don’t think that this pattern would work. If you configure your flows to run on local agent (the VM running 24/7), then scheduled flow runs will be picked up and executed there. From those flow runs, you could be spinning up some VMs, if this is possible to do programmatically with Wekeo, but Prefect has no way of delegating work to that machine, unless it’s registered as a Prefect agent, or a Dask cluster. The best approach I can see is to pre-provision all resources you need, and e.g. deploy a Kubernetes cluster on this cloud. Kubernetes could then take care of resource allocation and Prefect would orchestrate your flows and their states.

I was thinking of relying on labels to choose which VM will pick up the flow (as its planned to have multiple VMs and flows managed this way) Ill investigate more into Kubernetes clusters !

oh, that would definitely work, too, as long as those machines are always on and have a prefect agent started on them. However, you would have to take care of monitoring resources and ensuring that you assign flows to machines that have enough capacity, or spread the load in some other way (e.g. leveraging concurrency limits from Prefect Cloud). But this is probably the easiest way to get started.