How come there are “No Upcoming Runs” in the queue, when I’ve limited the “Flow Run Concurrency” to e.g.

10

but are plenty, when it is left to Unlimited? When there are no visible scheduled jobs, my agent polling that queue would not retrieve any jobs either 😕 As soon as I change it to

Unlimited

, it comes to life and starts running all the late jobs like nothing had happened. I’m using Prefect Cloud v2

Hmmm I wasn't able to reproduce this in prefect cloud, would you be able to provide some context around how the schedule was created? i.e. was it created through the UI or as part of the deployment? which type of schedule was used? does this occur with a new deployment/workqueue?

@Mason Menges I will create a minimally reproducible example and post the recipe here. Little tied up, hence I may need some time to get this done. To provide little bit of context 1.

prefect deployment build …

2. Some edits to the yaml e.g. schedule changed to interval 3.

prefect deployment apply …

All was well until the next day (I had shut down the agent for the night) and hence quite many schedules were late. 4. Started up agent and shut it down soon after (it started to replay all of late schedules) 5. Changed the flow concurrency in the queue (In UI) to 10 6. Started up agent and it run for a while after which I noticed no new logs rolling in 7. Queue on Prefect UI is empty.. this is where I started to look into the concurrency and discovered that toggling it off (set to Unlimited) brought back the expected functionality.

Awesome Thanks for the insight 😄

I’m sure there are plenty “late” schedules in the only queue I’ve got at the moment. Just tried to trigger the issue again by changing current

Unlimited

“Flow Run Concurrency (Optional)” setting to 10 and 20 and the problem is there as described earlier, but if I change it to 30, I can see 10 schedules with setting of 40, I see 20 schedules. I suppose you’d still require steps to reproduce this sort of situation?

Hi @jpuris - I looked into this issue and opened a ticket for it because I was seeing the same thing: https://github.com/PrefectHQ/prefect/issues/6898 After digging in a bit further, I realized that my work-queue had a load of pending runs attached to it that had never made it to running or completed. Can you try setting your date filter in the flow runs dashboard back to whenever you first started using 2.0 and select only the pending and running states and see what the run count is? If there are any runs in pending or running state, they could be using up the concurrency. It may be that yours is a separate issue but I'd like to see if it's connected.

@Jenny Exactly! I have exactly 20 runs in queue as pending! I suspect this may be because when the agent grabbed these scheduled runs I had ctrl+c’ed the process as I did not want it to run so many jobs at the time. Me terminating the agent process must have made them into “zombie” runs that the server expects to be fulfilled, but no agent is going to touch them and so on..

Yes exactly! I think you should be able to delete those runs and then your concurrency should fix itself (or just up the concurrency!)

I’ll consider this handled then and will track the github issue. Thank you @Jenny!

I am also experiencing this issue FWIW - we need a flow concurrency limit of exactly one due to external API limitations. Flows will get stuck in a late state even though the agent is not running them at all.

Hi @Rio McMahon - thanks for the feedback. Are you able to delete the runs that are stuck in pending?

@Jenny No flows show up as pending, they transition directly from

Scheduled

to

Late

. They are running on a 1 minute cadence. The agent polling for the queue never picks them up (confirming by looking at agent logs). Flows just continually pile up as

Late

without any ever being run.

Can you try setting your date filter in the flow runs dashboard back to whenever you first started using 2.0 and select only the pending and running states and see what the run count is? If there are any runs in pending or running state, they could be using up the concurrency. ☝️ This was the issue for both JP and I. If you have no pending runs then it's likely a separate issue and you're better off opening a new thread.

Looks like that was the issue - apologies for not reading the thread more clearly. Thanks!

Glad you got it cleared up! 😀