Hi I m running a prefect 2 5 0 agent on a VM in azure Occasi

Question

Hi I m running a prefect 2 5 0 agent on a VM in azure Occasionally when not running any jobs like in the middle of the night the agent throws the following exception when trying to ping the prefect api ```PrefectHTTPStatusError Client error 403 Forbidden for url <https api prefect cloud api accounts > lt ACCOUNTGUID gt wor kspaces lt WORKSPACEGUID gt work queues lt WORKQUEUEGUID gt get runs ``` At usr local lib python3 8 dist packages prefect client base py 125 Submitting any flows after this exception leads to them being visible in the Prefect cloud web interface as Scheduled or Late basically the agent completely dies after this Is this a known issue or could this be due to something else I m doing incorrectly

vholmer · Answer

Just realized this is probably very similar to < Khyaati Jindal> s issue

vholmer · Answer

Ideally they shouldn t throw this exception at all the agents shouldn t just stop working randomly Pragmatically however they shouldn t just throw this exception and keep existing in some kind of limbo requiring a manual restart I think it d be better if they just properly died allowing for e g a daemon to restart the agent automatically

Khuyen Tran · Answer

< vholmer> This looks like an bug to me Can you open an issue on so we can look a closer look at it

vholmer · Answer

< Khuyen Tran> Sure I just posted it

Robin Weiß · Answer

Hiya I am also seeing this on my side I had really weird pending and late flows before and now the 403s are also popping up

vholmer · Answer

< Robin Weiß> bump the github thread if possible so that the engineers that will work on it can see that multiple people are suffering from this in the issue itself slightly smiling face

vholmer · Answer

perhaps describe your case a bit more in depth too to add any potential further info

David Cheedie · Answer

Hi < vholmer> unrelated question to your bug but I m also running an agent on an azure VM ubuntu 20 04 and I m wondering how you keep the agent running as a background process It works if I run `prefect start agent test amp ` but if I don t clean logout with `exit` the session sends a hangup signal and kills the process

vholmer · Answer

< David Cheedie> I just run it as a service

David Cheedie · Answer

Exactly what I needed Thank you