Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello, I am running into an issue where my flows are not being scheduled as often as they should, leaving my work queue unhealthy. I'm using Kubernetes as my infrastructure, my agents are up and they are healthy. I can also confirm the queue is being polled actively.

I have a concurrency limit of 9, but I am only seeing one flow at a time being scheduled and having a "Running" state. No task limits are being used.
I'm on  v2.10.9.

How can I identify the cause of this bottleneck?

I have multiple jobs stuck in a "Late" and "Pending" state, they are eventually run one at a time, but some of them have been multiple hours (6+) late

I have 8 flow runs a day old stuck in a pending state, so it looks like that's the culprit, but other deployments of the same flow are able to run successfully in that 1 remaining slot