https://prefect.io logo
Title
m

Matt Delacour

04/30/2023, 7:56 PM
👋 I don't understand what changed in Prefect 2.0 cloud / agents recently. Everything was running smoothly but now when I start my agent, As soon as I strat a quick flow run, the work pool becomes unhealthy and I get the following error sometimes (most of the times I don't get any error).
does not have an infrastructure pid attached. Cancellation cannot be guaranteed.
It's very hard to have more logs than that so it's hard to understand the problem and debug. Any help would be helpful 🙏 Some more context • The agent is deployed in GCP VM • I use the exact same command line than specified in Cloud • Stopped working last week with Prefect 2.7.* on the agent side. Same problem with 2.10.*
And my work queue becomes unhealthy while the only thing I did was to start an agent ...
Don't know if it's related but some APIs don't work locally with a 404 error while the account and the workspace are the same as my agent
And same for starting an agent locally
y

Yaron Levi

04/30/2023, 9:12 PM
@Matt Delacour Maybe I'm wrong here, but aren't work pools feature still in beta?
m

Matt Delacour

04/30/2023, 11:24 PM
Maybe it's misleading but I used to spin up agents and doing the exact same command does not work anymore ... And today my agent cannot pull any jobs as it uses to be
And BTW, this is my startup script
And when using
PREFECT_LOGGING_LEVEL=DEBUG
It should be pulling jobs but nothing happen on my end. And my work queue still appears unhealthy
Well if I just create a new work queue, everything works as it used to be, So definitely something is off with the default work queue coming with Prefect cloud
Also my agent just died as, it seems like, it cannot reach out the cloud Same how can I debug that?
Work queues are so unreliable these days it's unbelievable .... Its has been working fine for months but keep breaking every couple hours 😢
👍 1
n

Nelson Griffiths

05/01/2023, 6:11 PM
I’m starting to see a lot of similar behavior with an agent going down every few hours. I’m going to get my traceback and share as well
m

Matt Delacour

05/01/2023, 7:49 PM
Thanks @Nelson Griffiths Happy to collaborate on that. Hopefully someone from Prefect will also look at it 🙏
n

Nelson Griffiths

05/01/2023, 8:46 PM
@Matt Delacour I came across this and it seems to have fixed my issues https://github.com/PrefectHQ/prefect/issues/7442
m

Matt Delacour

05/01/2023, 8:47 PM
Thanks for sharing
Opened this issue on Nov 5, 2022 · 35 comments
e

Ethan Veres

05/05/2023, 3:06 PM
@Matt Delacour I’m experiencing the same issues. I’m running my agent on ECS Fargate and my agent has become unreliable ever since prefect v2.10
were you able to resolve?
m

Matt Delacour

05/05/2023, 3:09 PM
Please add your comment in the issue mentioned above https://github.com/PrefectHQ/prefect/issues/7442 For some people people having the following ENV var set up fixes it
PREFECT_API_ENABLE_HTTP2=false prefect agent start
On my side it is still unreliable. Sometimes it works and sometimes it fails after a couple of hours 🤷
e

Ethan Veres

05/05/2023, 3:23 PM
ill try that env variable and see what it does. ill comment on that PR too
🙌 1