:wave:
I don't understand what changed in Prefect ...
# ask-communitym
Matt Delacour
04/30/2023, 7:56 PM👋
I don't understand what changed in Prefect 2.0 cloud / agents recently. Everything was running smoothly but now when I start my agent, As soon as I strat a quick flow run, the work pool becomes unhealthy and I get the following error sometimes (most of the times I don't get any error).
does not have an infrastructure pid attached. Cancellation cannot be guaranteed.It's very hard to have more logs than that so it's hard to understand the problem and debug. Any help would be helpful 🙏 Some more context • The agent is deployed in GCP VM • I use the exact same command line than specified in Cloud • Stopped working last week with Prefect 2.7.* on the agent side. Same problem with 2.10.*
➕ 1
Matt Delacour
04/30/2023, 7:59 PMAnd my work queue becomes unhealthy while the only thing I did was to start an agent ...
Matt Delacour
04/30/2023, 7:59 PM☝️ This error comes from the
cancel_runMatt Delacour
04/30/2023, 8:06 PMDon't know if it's related but some APIs don't work locally with a 404 error while the account and the workspace are the same as my agent
Matt Delacour
04/30/2023, 8:07 PMAnd same for starting an agent locally
y
Yaron Levi
04/30/2023, 9:12 PM@Matt Delacour Maybe I'm wrong here, but aren't work pools feature still in beta?
m
Matt Delacour
04/30/2023, 11:24 PMMaybe it's misleading but I used to spin up agents and doing the exact same command does not work anymore ...
And today my agent cannot pull any jobs as it uses to be
Matt Delacour
05/01/2023, 12:03 AMAnd BTW, this is my startup script
Matt Delacour
05/01/2023, 12:51 AMAnd when using
PREFECT_LOGGING_LEVEL=DEBUGMatt Delacour
05/01/2023, 1:58 AMWell if I just create a new work queue, everything works as it used to be,
So definitely something is off with the default work queue coming with Prefect cloud
Matt Delacour
05/01/2023, 12:05 PMAlso my agent just died as, it seems like, it cannot reach out the cloud
Same how can I debug that?
Matt Delacour
05/01/2023, 4:37 PMWork queues are so unreliable these days it's unbelievable ....
Its has been working fine for months but keep breaking every couple hours 😢
👍 1
n
Nelson Griffiths
05/01/2023, 6:11 PMI’m starting to see a lot of similar behavior with an agent going down every few hours. I’m going to get my traceback and share as well
m
Matt Delacour
05/01/2023, 7:49 PMThanks @Nelson Griffiths Happy to collaborate on that.
Hopefully someone from Prefect will also look at it 🙏
n
Nelson Griffiths
05/01/2023, 8:46 PM@Matt Delacour I came across this and it seems to have fixed my issues https://github.com/PrefectHQ/prefect/issues/7442
m
Matt Delacour
05/01/2023, 8:47 PMThanks for sharing
Matt Delacour
05/01/2023, 8:51 PMOpened this issue on Nov 5, 2022 · 35 comments
e
Ethan Veres
05/05/2023, 3:06 PM@Matt Delacour I’m experiencing the same issues. I’m running my agent on ECS Fargate and my agent has become unreliable ever since prefect v2.10
Ethan Veres
05/05/2023, 3:06 PMwere you able to resolve?
m
Matt Delacour
05/05/2023, 3:09 PMPlease add your comment in the issue mentioned above https://github.com/PrefectHQ/prefect/issues/7442
For some people people having the following ENV var set up fixes it
Copy code
PREFECT_API_ENABLE_HTTP2=false prefect agent startMatt Delacour
05/05/2023, 3:10 PMOn my side it is still unreliable. Sometimes it works and sometimes it fails after a couple of hours 🤷
e
Ethan Veres
05/05/2023, 3:23 PMill try that env variable and see what it does. ill comment on that PR too
🙌 1
5 Views