https://prefect.io logo
Title
a

Andreas Nord

05/31/2022, 1:33 PM
Hi! I'm running prefect agent with Azure VM, suggestions to save cost of the idle VM (I only need like 20 minutes nightly to run the flows)? Some built in functionality of Prefect that can help boot/shutdown to match with jobs
k

Kevin Kho

05/31/2022, 2:11 PM
The VM for the agent doesn’t need to be powerful. You can try agentless execution instead with Azure functions
prefect run --project project_name --name flow_name --execute
a

Andreas Nord

05/31/2022, 2:15 PM
Depends no? In my case I do the actual ETL in Python (for now)
k

Kevin Kho

05/31/2022, 2:31 PM
Ah I see, so for Local Agent, the agent runs the job, but you can also submit jobs to a Kubernetes cluster instead. In that case, the agent can be minimal. The
--execute
flag will run on the same machine. So if it fits in an Azure Function, maybe you can try that
a

Andreas Nord

05/31/2022, 2:37 PM
I am using DockerAgent but your point should be the same. My data volume is currently low so probably don't need a cluster, but I guess kubeternetes should scale better
k

Kevin Kho

05/31/2022, 2:38 PM
Yeah for AWS, we have the ECS agent which is a serverless container, but we don’t have an equivalent for Azure
a

Andreas Nord

05/31/2022, 2:44 PM
That would be useful, a tutorial with an example deployment on Azure with Prefect would be great to have at some point
k

Kevin Kho

05/31/2022, 2:50 PM
There is a write-up here
👀 1