Hey all :slightly_smiling_face: We are playing wi...
# prefect-community
r
Hey all 🙂 We are playing with the flow concurrency limits and added two labels with the following limits:
k8s
= 3 and
customer
= 1. So, we thought that with a prefect agent that has the labels
k8s
and
customer
, we should be able to run several flows at the same time (e.g. up to 3
k8s
labeled flows) but always only 1
customer
labeled flow. However, we were not able to run a
["customer", "k8s"]
and a
k8s
flow at the same time. 😮 We then added another
k8s
agent (without the
customer
label) and now the flows indeed can run add the same time, 1 flow on each agent. Is this intended behavior? For us that does not feel right, because in that case one needs to update infrastructure whenever adding flow concurrency limits, right? 🤔
n
Hi @Robin - A flow concurrency slot is allocated only if every passed label is below the set threshold. I'm surprised at the behavior you're seeing - is it the same without concurrency limits defined at all?
r
OK, that's the way we also intuitively understood the documentation. With below, you mean below or equal the threshold, right? Actually, if we just run two tasks, of which one has `["customer", "k8s"]`and one has
"k8s"
label, and we then increase the
customer
label to 2, the flows run in parallel again...
Which is also weird.
n
Is it possible that the runs you're creating have the
customer
label as well? We recently introduce flow run level labels that are set when the run is scheduled/created
(you can see these on the flow run page details tile)
r
hmm, not sure if I understand the question correctly. We set the flow label
customer
for the agent, for one of the flows and not for the other flow. We set the k8s label to all of them (both flows and the agent). Does that answer the question?
We also get some unexpected behavior for the task concurrency: We have several tasks in the flow that were supposed to run in parallel. We set the overall task concurrency to 100 and the "big data" tasks to 10. Additionally we set another task concurrency to 1. We expected that prefect would schedule the tasks such, that there are always up to 100 tasks running in parallel. Instead, it seems like only one mapped task can run at the time, such that only one task is run at the time, when the error log task map is executed... 🤔
n
sorry, I was unclear there @Robin - I was thinking for the individual runs you're creating - they should take the labels of the flow when the run is created but it sounds like you're doing some label modification after the fact that could be causing an issue:
(pictured is a flow with a
k8s
label but a run that I created that has both a
k8s
and
customer
label)
@Robin can you provide the signatures for your tasks there as well as how you're calling them? I have a suspicion for your task concurrency issue.