https://prefect.io logo
Title
v

vholmer

10/25/2022, 8:32 AM
Hi, I'm running a prefect 2.5.0 agent on a VM in azure. Occasionally when not running any jobs, like in the middle of the night, the agent throws the following exception when trying to ping the prefect api:
PrefectHTTPStatusError: Client error '403 Forbidden' for url
'<https://api.prefect.cloud/api/accounts/><ACCOUNTGUID>/wor
kspaces/<WORKSPACEGUID>/work_queues/<WORKQUEUEGUID>/get_runs'
At /usr/local/lib/python3.8/dist-packages/prefect/client/base.py:125. Submitting any flows after this exception leads to them being visible in the Prefect cloud web interface as "Scheduled" or "Late", basically the agent completely dies after this. Is this a known issue or could this be due to something else I'm doing incorrectly?
1
Just realized this is probably very similar to @Khyaati Jindal's issue.
Ideally they shouldn't throw this exception at all, the agents shouldn't just stop working randomly. Pragmatically, however, they shouldn't just throw this exception and keep existing in some kind of limbo requiring a manual restart. I think it'd be better if they just properly died, allowing for e.g. a daemon to restart the agent automatically
k

Khuyen Tran

10/25/2022, 3:55 PM
@vholmer This looks like an bug to me. Can you open an issue on Prefect GitHub so we can look a closer look at it?
v

vholmer

10/26/2022, 8:09 AM
@Khuyen Tran Sure! I just posted it: https://github.com/PrefectHQ/prefect/issues/7323
:gratitude-thank-you: 1
r

Robin Weiß

10/26/2022, 1:05 PM
Hiya! I am also seeing this on my side. I had really weird pending and late flows before and now the 403s are also popping up
v

vholmer

10/26/2022, 1:15 PM
@Robin Weiß bump the github thread if possible, so that the engineers that will work on it can see that multiple people are suffering from this in the issue itself 🙂
perhaps describe your case a bit more in-depth too, to add any potential further info
d

David Cheedie

11/09/2022, 2:39 PM
Hi @vholmer, unrelated question to your bug but I'm also running an agent on an azure VM - ubuntu 20.04 - and I'm wondering how you keep the agent running as a background process? It works if I run
prefect start agent 'test' &
but if I don't clean logout with
exit
the session sends a hangup signal and kills the process
v

vholmer

11/09/2022, 2:49 PM
d

David Cheedie

11/09/2022, 3:01 PM
Exactly what I needed. Thank you!
👍 1