Hi, I'm running a prefect 2.5.0 agent on a VM in azure. Occasionally when not running any jobs, like in the middle of the night, the agent throws the following exception when trying to ping the prefect api:

PrefectHTTPStatusError: Client error '403 Forbidden' for url
'<https://api.prefect.cloud/api/accounts/><ACCOUNTGUID>/wor
kspaces/<WORKSPACEGUID>/work_queues/<WORKQUEUEGUID>/get_runs'

At /usr/local/lib/python3.8/dist-packages/prefect/client/base.py:125. Submitting any flows after this exception leads to them being visible in the Prefect cloud web interface as "Scheduled" or "Late", basically the agent completely dies after this. Is this a known issue or could this be due to something else I'm doing incorrectly?

@vholmer This looks like an bug to me. Can you open an issue on Prefect GitHub so we can look a closer look at it?

@Khuyen Tran Sure! I just posted it: https://github.com/PrefectHQ/prefect/issues/7323

Hiya! I am also seeing this on my side. I had really weird pending and late flows before and now the 403s are also popping up

@Robin Weiß bump the github thread if possible, so that the engineers that will work on it can see that multiple people are suffering from this in the issue itself 🙂

Hi @vholmer, unrelated question to your bug but I'm also running an agent on an azure VM - ubuntu 20.04 - and I'm wondering how you keep the agent running as a background process? It works if I run

prefect start agent 'test' &

but if I don't clean logout with

exit

the session sends a hangup signal and kills the process

@David Cheedie I just run it as a service: https://discourse.prefect.io/t/how-to-run-a-prefect-2-agent-as-a-systemd-service-on-linux/1450/1

Exactly what I needed. Thank you!