Hello ! We are currently blocked by the fact that ...
# prefect-server
Hello ! We are currently blocked by the fact that our flows are stuck in the
state, while these runs have no labels associated with a concurrency limit In addition, we indeed have an agent with the matching label, but nothing is happening Note that some of the first runs were able to be executed (less than 10) Any idea maybe ? 😕
One of the stuck runs:
There are many potential reasons but the two you mentioned are definitely the most common ones. Can you cross check if: 1. Your agent has the label “production” assigned? 2. Your agent is healthy and running? 3. Your agent is overwhelmed - can you check the agent logs and also send a screenshot from the Agents page in the UI to see how many scheduled late runs this agent has? Clearing the late runs can help here 4. Your API key attached to the agent is still valid? Based on the fact that 10 runs were successful and the rest got stuck, it looks like some issue with the agent. You could check if restarting the agent helps here
Hello @Anna Geller, 1. Yes 2. Yes, but I restarted it just to be sure, still not picking anything 3. 900 late runs, according to logs the agent stopped picking up anything one hour ago, for no particular reason. We have been load testing this configuration for over one week now, and have not encountered problems on agents level. Clearing the late runs can help for sure, but we would rather avoid doing that as it's removing symptoms without understanding the cause, it may reproduce as 900 concurrent runs can happen again. 4. The API key is set on "never expire" from service accounts page
ok, so the issue is that the scheduler is struggling with more than 750 flow runs that are scheduled at the same time - to the best of my knowledge, clearing the late runs is the only way to unblock your agent