[Beginners questions] [VM coordination with Prefect Cloud] [Work pools] [Agents] Hey there, I successfully deployed a prefect flow and it perfectly pulls data from an API and uploads it to GCS and then to BigQuery. But when it comes to scheduling I am missing understanding. I use a VM from Google and host my code on GitHub. When I am running the flow from my VMs CLI: 1. I start the default work pool via

prefect agent start -q 'default'

2. I call my flow

python my_flow_file.py

I would like my flow now to run independently without me needing to do (1) and (2). I scheduled my flow to run every Saturday, but when I checked, today, it was in status

late

and was waiting for a agent to pick up the run. How would I start an agent when I am not in VS code, starting it via the CLI? I assumed it would be started via the Prefect Cloud or my VM that was running.

run the agent under systemd on your vm, https://discourse.prefect.io/t/how-to-run-a-prefect-2-agent-as-a-systemd-service-on-linux/1450

Thanks @redsquare. Is this how it is intended to be? I am just not clear with why this function is not something built-in or more automated? Isn't this a valid use case, or even THE main use case (running an independent pipeline)?

depends, some folk just want agents spinning up on demand, run flows then quit, others run in k8s where it is not needed. I am sure in the future they will offer the creation under systemd but its not that hard and perhaps not the main use case these days

@redsquare, don't you still need an agent running to handle your kubernetes job submissions? Or do you mean the agent would be set to run on a pod in your kubernetes cluster?

Correct the agent is a pod with health check etc and cycles when needed

Thanks for that, it just did not make sense that you can schedule a run but then need a worker that requieres to be started seperatley. It would, as a newbie, be more intuitive to get the worker automatically if i schedule something - which implies that i want that job to run automatically?! @redsquare I just realized my VM uses Ubuntu and I am already struggeling with the first step

cd /etc/systemd/system

- is there an ubuntu adaption or can you point out what the equivalent steps would be?

systemd should be installed on most all linux distros. Some configuration info for it on ubuntu is here, if that’s helpful.

For transparency, in case if someone has the same issue, on my VM nothing mentioned under your link is visible in my VM.

which ubuntu are you running