Hi everyone, I want to ask about flow concurrency: I’ve set flow_concurrency for label `agent:k8s-pr...

Ranu Goldan

06/03/2021, 1:22 PM

Hi everyone, I want to ask about flow concurrency: I’ve set flow_concurrency for label

agent:k8s-prod

to 100 But I’ve seen the maximum of concurrent run is only 22-24, the other runs is scheduled I have 1 kubernetes agent deployment on my k8s cluster How to make the flow run concurrency more than 24? Anything I miss? Thanks in advance

Ranu Goldan

06/03/2021, 1:25 PM

It doesn’t even running more scheduled flow run when half done

Ranu Goldan

06/03/2021, 1:28 PM

Update: it suddenly incresed to 50

Ranu Goldan

06/03/2021, 1:29 PM

Like about 40~ started to run in the same time. Maybe any logic behind this, how is the behavior of scheduled runs?

Kevin Kho

06/03/2021, 1:30 PM

Hey @Ranu Goldan, I don't know this. I'll ask a team member and get back to you

Ranu Goldan

06/03/2021, 1:51 PM

Thanks @Kevin Kho!

nicholas

06/03/2021, 2:13 PM

Hi @Ranu Goldan - how is your k8s cluster configured? It sounds like you're running into resource constraints somewhere, perhaps your cluster is starved?

Ranu Goldan

06/03/2021, 11:22 PM

I use GKE standard with n1-standard-4 6 node with default configurations, i’m not sure either about the constraint but i just left it with default while creating the cluster. And I test those above while my cluster is empty, no other jobs

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.