https://prefect.io logo
Title
j

Joshua Greenhalgh

05/30/2022, 11:45 AM
If a want to run a large backfill on k8s using the framework defined here https://discourse.prefect.io/t/i-want-to-use-parameter-values-for-backfilling-but-prefe[…]ter-defaults-provided-at-registration-how-to-solve-that/49 - is there a way to batch things up so I am only ever running say 10 jobs at a time?
a

Anna Geller

05/30/2022, 11:47 AM
You can set a flow-level concurrency limit of 10 to configure that
j

Joshua Greenhalgh

05/30/2022, 11:47 AM
Is that a cloud setting?
a

Anna Geller

05/30/2022, 11:47 AM
yes
j

Joshua Greenhalgh

05/30/2022, 11:50 AM
Not on the free tier though? Ok well will have to upgrade for this I think! If I never go over the 20k tasks per month it's still free yeah?
a

Anna Geller

05/30/2022, 12:02 PM
Standard tier is also a free tier
I would always recommend Standard tier, Starter is really meant for nothing more than a PoC
j

Joshua Greenhalgh

05/30/2022, 12:04 PM
How does the tag interface with the agent tag? If my agent has tag ['a'] and I set a limit on tag ['b'] will my flow still get picked up by the agent if I give it tags ['a','b'] and the concurrency limit for ['b'] will be respected?
have changed to standard!
Or does my agent need to have all the tags ['a','b']?
I never upgraded since I am the only DE so no need for other users
but this is a good enough reason to do so!
a

Anna Geller

05/30/2022, 12:19 PM
if you set the concurrency limit on flow label "b" and your agent has the same label "b" assigned when starting the agent, then there is a match and Prefect will deploy only max. 10 runs of this flow to this matching agent
j

Joshua Greenhalgh

05/30/2022, 12:20 PM
Ok so for every concurrency limit I setup I need to add a new tag to the agent? But the flow itself only needs to have some subset of the ones on the agent?
a

Anna Geller

05/30/2022, 12:23 PM
correct, the agent can have more labels than your flow's run config does
j

Joshua Greenhalgh

05/30/2022, 12:23 PM
Perfect! Thanks!
👍 1