Our Prefect agent on Prefect 2 has stopped working...
# ask-community
a
Our Prefect agent on Prefect 2 has stopped working and no longer seems to be receiving any communication from the Prefect server. The code warns that the agent will be deprecated in Sep 2024 but I was told that an agent will still work with Prefect cloud as long as we keep our agent running on Prefect 2. Is that no longer accurate? Is there a maximum Prefect 2 minor version in which the agent will still work? Or has Prefect Cloud totally stopped communicating with prefect agents?
n
hi @Austin Weisgrau - agents should still function in the 2.x lineage (all 2.x versions) can you be more specific about what you're seeing / the setup you have?
upvote 1
a
ok word - I've been having a lot of issues with the prefect agent lately that feel hard to diagnose. The agent will stop communicating with Prefect Cloud, restarting the agent can help, sometimes there seems to be a concurrency limit on how many flows the agent will kick off even though my config doesn't specify any concurrency limits. All of these issues started after September 2024 after a year+ of no issues and without any changes to our config. I'm feeling like I need to migrate to Prefect 3 in order to have reliability
n
iirc there are several memory-consumption related improvements with workers that haven't been backported to agents, so is it possible that your agent process is OOMing? or how is your agent process running / what infra is it kicking off runs on?
any logs from the agent process may be helpful
a
it spins up separate AWS ECS tasks to kick off flow runs, so it's just the one agent process running with no flows running in that container
it does seem to stream logs from the jobs it kicks off
A prefect flow will hang somewhere, and basically permanently hang and it won't crash or do anything. the parent agent seems to hang as well when this happens. I don't think OOM would explain it, because I'd expect OOM to cause a crash I'm not sure if there is a timeout on Prefect Cloud API calls, but it seems like it might be hanging when a task is starting/finishing and trying to communicate with Prefect Cloud
OK I got migrated over to Prefect 3 and ecs push work pools and such, hopefully will see more stability