If a want to run a large backfill on k8s using the...
# ask-community
j
If a want to run a large backfill on k8s using the framework defined here https://discourse.prefect.io/t/i-want-to-use-parameter-values-for-backfilling-but-prefe[…]ter-defaults-provided-at-registration-how-to-solve-that/49 - is there a way to batch things up so I am only ever running say 10 jobs at a time?
a
You can set a flow-level concurrency limit of 10 to configure that
j
Is that a cloud setting?
a
yes
j
Not on the free tier though? Ok well will have to upgrade for this I think! If I never go over the 20k tasks per month it's still free yeah?
a
Standard tier is also a free tier
I would always recommend Standard tier, Starter is really meant for nothing more than a PoC
j
How does the tag interface with the agent tag? If my agent has tag ['a'] and I set a limit on tag ['b'] will my flow still get picked up by the agent if I give it tags ['a','b'] and the concurrency limit for ['b'] will be respected?
have changed to standard!
Or does my agent need to have all the tags ['a','b']?
I never upgraded since I am the only DE so no need for other users
but this is a good enough reason to do so!
a
if you set the concurrency limit on flow label "b" and your agent has the same label "b" assigned when starting the agent, then there is a match and Prefect will deploy only max. 10 runs of this flow to this matching agent
j
Ok so for every concurrency limit I setup I need to add a new tag to the agent? But the flow itself only needs to have some subset of the ones on the agent?
a
correct, the agent can have more labels than your flow's run config does
j
Perfect! Thanks!
👍 1