Hi Prefect team, As of last week we've seen flow runs randomly get stuck in the

Pending

state. Besides preventing the flow from running this prevents the next scheduled run from happening, since we use concurrency to restrict each of our flows to one run at a time. There's no obvious relationship between the flows getting stuck, it seems random. Since we've moved to push pools there are also no logs to diagnose the issue with, meaning most of the advice here is not relevant https://discourse.prefect.io/t/my-flow-run-in-prefect-2-is-stuck-in-a-pending-state-what-can-i-do/2012. Any ideas what could be causing this issue? I'm happy to share account details 1-on-1 so y'all can investigate further.

👋 hey! If you could share your workspace id and a (or a few) stuck flow run ids, I can take a look. Feel free to DM it if you'd prefer

@Robert Banick I am encountering a similar issue. Opened a ticket: https://github.com/PrefectHQ/prefect/issues/11499

@Yaron Levi I spoke 1v1 w/ @Jake Kaplan about this yesterday and Prefect identified an issue on their end. Seems like a fix is coming today

hey, this went out yesterday and should be resolved. Please don't hesitate to respond here or on the issue if you see runs still stuck in PENDING going forward. Thanks to you both for flagging!