Hi, I'm running a prefect 2.5.0 agent on a VM in a...
# prefect-community
v
Hi, I'm running a prefect 2.5.0 agent on a VM in azure. Occasionally when not running any jobs, like in the middle of the night, the agent throws the following exception when trying to ping the prefect api:
Copy code
PrefectHTTPStatusError: Client error '403 Forbidden' for url
'<https://api.prefect.cloud/api/accounts/><ACCOUNTGUID>/wor
kspaces/<WORKSPACEGUID>/work_queues/<WORKQUEUEGUID>/get_runs'
At /usr/local/lib/python3.8/dist-packages/prefect/client/base.py:125. Submitting any flows after this exception leads to them being visible in the Prefect cloud web interface as "Scheduled" or "Late", basically the agent completely dies after this. Is this a known issue or could this be due to something else I'm doing incorrectly?
1
Just realized this is probably very similar to @Khyaati Jindal's issue.
Ideally they shouldn't throw this exception at all, the agents shouldn't just stop working randomly. Pragmatically, however, they shouldn't just throw this exception and keep existing in some kind of limbo requiring a manual restart. I think it'd be better if they just properly died, allowing for e.g. a daemon to restart the agent automatically
k
@vholmer This looks like an bug to me. Can you open an issue on Prefect GitHub so we can look a closer look at it?
v
@Khuyen Tran Sure! I just posted it: https://github.com/PrefectHQ/prefect/issues/7323
gratitude thank you 1
r
Hiya! I am also seeing this on my side. I had really weird pending and late flows before and now the 403s are also popping up
v
@Robin Weiß bump the github thread if possible, so that the engineers that will work on it can see that multiple people are suffering from this in the issue itself 🙂
perhaps describe your case a bit more in-depth too, to add any potential further info
d
Hi @vholmer, unrelated question to your bug but I'm also running an agent on an azure VM - ubuntu 20.04 - and I'm wondering how you keep the agent running as a background process? It works if I run
prefect start agent 'test' &
but if I don't clean logout with
exit
the session sends a hangup signal and kills the process
v
d
Exactly what I needed. Thank you!
👍 1