Hi all 🖐️ I'm having an issue with an agent connected to Prefect Cloud. The agent is listening to a queue named _agent_1._ The concurrency limit for the queue is 4. Everything worked as expected for many weeks, but recently the agent stopped receiving new runs. The it started receiving runs again after I increased the concurrency limit to 100. I know that sometimes this happens when there are "ghost" runs - runs in Pending or Running state that are not really being executed by an agent, but they contribute to "used" concurrency number. But this time it is different. There are no Pending or Running runs and the total count of runs in non-finished state (mostly crashed) is about 20. But processing starts only when I set the concurrency limit to a value that is way higher than 20 (above 100). My question is, what causes this and how can it be fixed and avoided? Apparently, Orion/Prefect Cloud thinks the concurrency limit is reached, but I don't understand why. Any help greatly appreciated 🙏

Hi! Thanks for the report. Is this only happening with Prefect Cloud? There’s a similar issue at https://github.com/PrefectHQ/prefect/issues/8238

Hi! I’d like to look into this, do you mind dm-ing me your workspace id where this issue is happening?