Has anyone come across issues where a workers concurrency li Prefect Community #ask-community

Has anyone come across issues where a workers conc...

Joe

01/21/2025, 2:02 AM

Has anyone come across issues where a workers concurrency limit. I'm getting this "Flow run limit reached; 5 flow runs in progress." in the worker logs, but it seems that this worker does not have any tasks running on it from what I can tell.

Joe

01/21/2025, 2:03 AM

Along with that message I get a few of these afterward "Aborted submission of flow run 'e5e69476-8109-4bab-9985-8e34dec1881f'. Server sent an abort signal: Cannot transition a flow run with in-process retries to PENDING state."

Joe

01/21/2025, 2:04 AM

I feel like these issues only started after upgrading to 3.1.13

Joe

01/21/2025, 2:05 AM

I'm only getting through max 4 flows across 6 workers that each have a concurrency limit of 5.

Joe

01/21/2025, 5:40 AM

hmm i went through and cleaned up all the old flows in prefect UI and they now seem to be getting back up to the limit

Bianca Hoch

01/21/2025, 9:34 PM

Hi Joe! Based on what you've described, there's a chance that there were a few flow runs stuck in a transitive state that were blocking your concurrency limit. If this happens again, I'd recommend going to the work pool the runs are submitted to, and filtering for any runs stuck in

Pending

Running

, or

Cancelling

. If you want to be alerted of hanging flow runs, you could set up an automation to send you an email notification if any run stays in the aforementioned states for longer than necessary (ie: more than 10 hours, 24 hours, etc.)

Joe

01/21/2025, 11:41 PM

Thanks Bianca, It seems that was the case. I'm looking into the new feature released in 3.1.8 to use heart beats to monitor when flows need to be marked as timed out. Currently my workers get restarted and flows get stuck as "running" when they've actually got no worker running them. Thanks

19 Views

Open in Slack

Previous Next