https://prefect.io logo
Title
i

Igor Kotua

10/04/2022, 7:32 AM
Hi, I have a question about concurrency. I have a queue with concurrency = 1. Yesterday I manually started a job run via UI and also I have scheduled the same flow to run in 10 hours or so. The manually triggered job didn't finished in 10 hours, so another one shouldn't have started (because concurrency = 1), but it did start. Am I doing something wrong?
c

Christopher Boyd

10/04/2022, 2:26 PM
Is this Prefect 1 or Prefect 2? Are you using tags? How is your concurrency configured?
i

Igor Kotua

10/04/2022, 2:33 PM
Prefect Cloud. Concurrency configured on a queue level - I have a queue dedicated only for 1 flow. For this queue I set concurrency = 1
Also I added tag limiting concurrency to 1 for each task inside this flow - also didn't help
c

Christopher Boyd

10/04/2022, 2:35 PM
prefect cloud is the cloud offering, not necessarily 1 or 2
there is prefect cloud 1 and prefect cloud 2
i

Igor Kotua

10/04/2022, 2:38 PM
prefect cloud 2
c

Christopher Boyd

10/04/2022, 2:43 PM
Did you submit the run manually with tags? What about the scheduled one? The concurrency limit should be set on ‘matching’ flows and runs, where matching is decided based on the tags submitted to the agent
i

Igor Kotua

10/04/2022, 2:55 PM
Tags are written in code, so I guess they are always there
c

Christopher Boyd

10/04/2022, 2:55 PM
when you look at the flow run history, are you able to see the tags in the logs when the flow is submitted
I’m trying to determine when you submit a manual run versus a scheduled run if the tagset matched, which is what the concurrency limit should be keying off of
if they were submitted ot the same agent.
i

Igor Kotua

10/04/2022, 2:59 PM
Hmm, but when I set concurrency (1) limit on a whole queue (and all runs are inside this queue), should I also "match" tags?
c

Christopher Boyd

10/04/2022, 3:00 PM
yes
https://docs.prefect.io/concepts/work-queues/#work-queue-concurrency
For example, a work queue with a concurrency limit of 5 will only release new work if fewer than 5 matching runs are currently in a Running or Pending state. If 3 runs are Running or Pending, polling the queue for work will only result in 2 new runs, even if there are many more available, to ensure that the concurrency limit is not exceeded.
The matching runs are based on tags, not necessarily the queue itself
i

Igor Kotua

10/04/2022, 3:04 PM
I don't understand, sorry
I have a queue concurrency limit of 1, one task have been running, second task still started (it was autoscheduled)
By task I mean flow here (couldn't set tags on flows as far as I see)
Here is a picture:
Liberal-grebe was started manually at 6:55 p, yesterday
It was running 17 hours before failing, so up until 12 am today. Teal-calm started at 5 am today, while another flow has been running
And as you see flow run concurrency is 1 on the picture
Previously flow run concurrency was working fine for me
May be the problem is because this run was manually triggered?
Sorry for so much text, really appreciate your time looking into this
c

Christopher Boyd

10/04/2022, 3:26 PM
When you click into the flow-run for each, what are the tags assigned - e.g.:
i

Igor Kotua

10/04/2022, 3:47 PM
One has a tag "autoscheduled"
Another has no tags
c

Christopher Boyd

10/04/2022, 4:00 PM
That would be the discrepancy
The concurrency is matching tags
if the tags don’t match, concurrency is not enforced as it’s unaware that it needs to be enforced
i

Igor Kotua

10/04/2022, 4:08 PM
Thanks!
But to be honest, it's a bit strange behaviour