https://prefect.io logo
Join Slack
Powered by
I can't seem to get Concurrency limits to actually...
# ask-community
e
I can't seem to get Concurrency limits to actually work. Self hosted Prefect 2.20.16. I have one (EC2/Docker) work pool with one work queue named default. Everything runs OK normally. I've tried setting Concurrency limits under 
Concurrency
, 
Work Pool > Flow Run Concurrency
, and 
Work Pool > Work Queue > Flow Run Concurrency
but any number I put, all jobs get stuck in a Late state even with nothing actively running, whether I put 3 or 30. I remove the limit and Flows immediately go to Pending and Running. Having read the docs I'm not seeing anything misconfigured... I just dont know why the limit is blocking any Flow from starting. Any ideas?
b
Hi Eric! I'd recommend checking for older flow runs that are stuck in a 
Running
, 
Pending
, or `Cancelling`state. If you have a backlog of runs in those states, they may be occupying concurrency slots and not releasing them.
e
Ah no luck with that, nothing backlogged
Oh hmm. I've been trying using 3 as a limit with no luck. Upped that to 100 and then jobs started. So I think you may be on to something w/ other jobs backlogging, but I can't seem to find what/how
Screenshot 2025-03-14 at 11.38.05 AM.png
ok so Flows won't run anything up to 19. I change it to 20 and then jobs finally run 🤔. So there does seem to be 19 jobs that are considered running for concurrency purposes but not seeing anything in an active state. Deleted all previous 
Cancelled
runs but no luck
b
hmmm.. 19 is such an odd number (literally). You're only setting the concurrency limit at the work queue level, correct? No concurrency limits have been applied to a worker, work pool, or even at the deployment level?
e
Yes very odd (lol). Only at the queue level. Tried setting limits at the other places but the same behavior
b
To keep the troubleshooting train rolling, here's a script that may help rule out the possibility of hidden flow runs that are occupying those concurrency slots. At a high level, the script look for flow runs stuck in 
BAD_STATES
, and then deletes those flow runs using the Prefect client.
My thinking is you can add a few more states to the 
BAD_STATES
list (for example, 
Pending
) and just try running the 
list_flow_runs_with_state
function first, without deleting anything just to see what it returns.
32 Views
Open in Slack
PreviousNext
Powered by