Hello, in my prefect cloud account, i have set concurrency on a workpool. However, I'm seeing some odd behavior; When I set the concurrency to 20, even when there are only a couple flow runs executing i get a bunch that are stuck in pending state. And when I raise it up to 25 I never see issues. I'm wondering if there are left over artfiacts that might be causing this in my account that I cannot see

Hey Eric! Hmm..if raising the concurrency limit allows more work to pass through, you may have some flow runs that are taking up concurrency slots. I'd recommend checking for flows that are stuck in

Running

,

Cancelling

, in addition to

Pending

. If they're old, you'll want to ensure they are either

Cancelled

or deleted to free up those slots.

So I’ve already done that. The flow runs are getting started ok, but I’m not sure why tasks are getting stuck

Huh. Work pool concurrency applies to flow run execution, it shouldn't have any influence on individual task runs 🤔 It could be worth enabling

DEBUG

level logs in this case, at least for a flow where you see tasks getting stuck quite often. You can set it as an environment variable like so:

export PREFECT_LOGGING_LEVEL=DEBUG

Side note, what kind of infrastructure are you using for your flows?

I have an aws eks cluster set up with a prefect worker pod running on it

Gotcha. I'd recommend checking the worker logs as well, if you haven't already. There could be some indication of what could be going on infrastructure-wise that is causing this.

I was having a similar related issue: My tasks were not starting and they were failing with

"received abort during orchestration: The enclosing flow must be running to begin task execution. Task run is in PENDING state"

. Turns out i had 2 flows were task were labeled as "retrying" and those were filling up my concucrrency_slots. Thanks for the clues from this thread we were able to solve it by resertting the concurrecnty slos in the configuration