Hi, I'm running a prefect 2.5.0 agent on a VM in azure. Occasionally when not running any jobs, like...

vholmer

10/25/2022, 8:32 AM

Hi, I'm running a prefect 2.5.0 agent on a VM in azure. Occasionally when not running any jobs, like in the middle of the night, the agent throws the following exception when trying to ping the prefect api:

Copy code

PrefectHTTPStatusError: Client error '403 Forbidden' for url
'<https://api.prefect.cloud/api/accounts/><ACCOUNTGUID>/wor
kspaces/<WORKSPACEGUID>/work_queues/<WORKQUEUEGUID>/get_runs'

At /usr/local/lib/python3.8/dist-packages/prefect/client/base.py:125. Submitting any flows after this exception leads to them being visible in the Prefect cloud web interface as "Scheduled" or "Late", basically the agent completely dies after this. Is this a known issue or could this be due to something else I'm doing incorrectly?

✅ 1

vholmer

10/25/2022, 1:36 PM

Just realized this is probably very similar to @Khyaati Jindal's issue.

vholmer

10/25/2022, 1:44 PM

Ideally they shouldn't throw this exception at all, the agents shouldn't just stop working randomly. Pragmatically, however, they shouldn't just throw this exception and keep existing in some kind of limbo requiring a manual restart. I think it'd be better if they just properly died, allowing for e.g. a daemon to restart the agent automatically

Khuyen Tran

10/25/2022, 3:55 PM

@vholmer This looks like an bug to me. Can you open an issue on Prefect GitHub so we can look a closer look at it?

vholmer

10/26/2022, 8:09 AM

@Khuyen Tran Sure! I just posted it: https://github.com/PrefectHQ/prefect/issues/7323

gratitude thank you 1

Robin Weiß

10/26/2022, 1:05 PM

Hiya! I am also seeing this on my side. I had really weird pending and late flows before and now the 403s are also popping up

vholmer

10/26/2022, 1:15 PM

@Robin Weiß bump the github thread if possible, so that the engineers that will work on it can see that multiple people are suffering from this in the issue itself 🙂

vholmer

10/26/2022, 1:16 PM

perhaps describe your case a bit more in-depth too, to add any potential further info

David Cheedie

11/09/2022, 2:39 PM

Hi @vholmer, unrelated question to your bug but I'm also running an agent on an azure VM - ubuntu 20.04 - and I'm wondering how you keep the agent running as a background process? It works if I run

prefect start agent 'test' &

but if I don't clean logout with

exit

the session sends a hangup signal and kills the process

vholmer

11/09/2022, 2:49 PM

@David Cheedie I just run it as a service: https://discourse.prefect.io/t/how-to-run-a-prefect-2-agent-as-a-systemd-service-on-linux/1450/1

🙏 2

David Cheedie

11/09/2022, 3:01 PM

Exactly what I needed. Thank you!

👍 1

6 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.