https://prefect.io
Join Slack
:wave: I don't understand what changed in Prefect 2.0 cloud / agents recently. Everything was runnin...
# ask-community
m
👋 I don't understand what changed in Prefect 2.0 cloud / agents recently. Everything was running smoothly but now when I start my agent, As soon as I strat a quick flow run, the work pool becomes unhealthy and I get the following error sometimes (most of the times I don't get any error).
does not have an infrastructure pid attached. Cancellation cannot be guaranteed.
It's very hard to have more logs than that so it's hard to understand the problem and debug. Any help would be helpful 🙏 Some more context • The agent is deployed in GCP VM • I use the exact same command line than specified in Cloud • Stopped working last week with Prefect 2.7.* on the agent side. Same problem with 2.10.*
➕ 1
And my work queue becomes unhealthy while the only thing I did was to start an agent ...
Don't know if it's related but some APIs don't work locally with a 404 error while the account and the workspace are the same as my agent
And same for starting an agent locally
y
@Matt Delacour Maybe I'm wrong here, but aren't work pools feature still in beta?
m
Maybe it's misleading but I used to spin up agents and doing the exact same command does not work anymore ... And today my agent cannot pull any jobs as it uses to be
And BTW, this is my startup script
And when using 
PREFECT_LOGGING_LEVEL=DEBUG
It should be pulling jobs but nothing happen on my end. And my work queue still appears unhealthy
Well if I just create a new work queue, everything works as it used to be, So definitely something is off with the default work queue coming with Prefect cloud
Also my agent just died as, it seems like, it cannot reach out the cloud Same how can I debug that?
Work queues are so unreliable these days it's unbelievable .... Its has been working fine for months but keep breaking every couple hours 😢
👍 1
n
I’m starting to see a lot of similar behavior with an agent going down every few hours. I’m going to get my traceback and share as well
m
Thanks @Nelson Griffiths Happy to collaborate on that. Hopefully someone from Prefect will also look at it 🙏
n
@Matt Delacour I came across this and it seems to have fixed my issues https://github.com/PrefectHQ/prefect/issues/7442
m
Thanks for sharing
Opened this issue on Nov 5, 2022 · 35 comments
e
@Matt Delacour I’m experiencing the same issues. I’m running my agent on ECS Fargate and my agent has become unreliable ever since prefect v2.10
were you able to resolve?
m
Please add your comment in the issue mentioned above https://github.com/PrefectHQ/prefect/issues/7442 For some people people having the following ENV var set up fixes it
Copy code
PREFECT_API_ENABLE_HTTP2=false prefect agent start
On my side it is still unreliable. Sometimes it works and sometimes it fails after a couple of hours 🤷
e
ill try that env variable and see what it does. ill comment on that PR too
🙌 1
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by