https://prefect.io
Join Slack
I can't seem to get Concurrency limits to actually work. Self hosted Prefect 2.20.16. I have one (...
# ask-community
e
I can't seem to get Concurrency limits to actually work. Self hosted Prefect 2.20.16. I have one (EC2/Docker) work pool with one work queue named default. Everything runs OK normally. I've tried setting Concurrency limits under 
Concurrency
, 
Work Pool > Flow Run Concurrency
, and 
Work Pool > Work Queue > Flow Run Concurrency
but any number I put, all jobs get stuck in a Late state even with nothing actively running, whether I put 3 or 30. I remove the limit and Flows immediately go to Pending and Running. Having read the docs I'm not seeing anything misconfigured... I just dont know why the limit is blocking any Flow from starting. Any ideas?
b
Hi Eric! I'd recommend checking for older flow runs that are stuck in a 
Running
, 
Pending
, or `Cancelling`state. If you have a backlog of runs in those states, they may be occupying concurrency slots and not releasing them.
e
Ah no luck with that, nothing backlogged
Oh hmm. I've been trying using 3 as a limit with no luck. Upped that to 100 and then jobs started. So I think you may be on to something w/ other jobs backlogging, but I can't seem to find what/how
Screenshot 2025-03-14 at 11.38.05 AM.png
ok so Flows won't run anything up to 19. I change it to 20 and then jobs finally run 🤔. So there does seem to be 19 jobs that are considered running for concurrency purposes but not seeing anything in an active state. Deleted all previous 
Cancelled
runs but no luck
b
hmmm.. 19 is such an odd number (literally). You're only setting the concurrency limit at the work queue level, correct? No concurrency limits have been applied to a worker, work pool, or even at the deployment level?
e
Yes very odd (lol). Only at the queue level. Tried setting limits at the other places but the same behavior
b
To keep the troubleshooting train rolling, here's a script that may help rule out the possibility of hidden flow runs that are occupying those concurrency slots. At a high level, the script look for flow runs stuck in 
BAD_STATES
, and then deletes those flow runs using the Prefect client.
My thinking is you can add a few more states to the 
BAD_STATES
list (for example, 
Pending
) and just try running the 
list_flow_runs_with_state
function first, without deleting anything just to see what it returns.
23 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by