Nikita Samoylov
12/17/2021, 11:19 AMAnna Geller
$ prefect agent local start --no-hostname-label --label prod1
$ prefect agent local start --no-hostname-label --label prod2
Currently, Prefect has no notion of a task queue that would allow load-balancing flow runs across multiple agents based on available resources on each agent. If you need such functionality, you would need to assign unique labels to your agents, and then assign corresponding labels to your flows to spread the load across multiple agents. For instance:
Flow 1:
from prefect.run_configs import UniversalRun
...
with Flow(name="example1", run_config=UniversalRun(labels=["prod1"])) as flow:
Flow 2:
from prefect.run_configs import UniversalRun
...
with Flow(name="example2", run_config=UniversalRun(labels=["prod2"])) as flow:
To help with scale, we recommend using one of the other agents, such as:
• KubernetesAgent
• ECSAgent
• VertexAgentNikita Samoylov
12/17/2021, 11:49 AMprod1
agent is shut down for some reason the flows scheduled for this prod1
will not be executed at all. How do you handle it?
2. We have mapped tasks in our flows. If mapped task is split to 100 tasks it will be counted as 100 separate successfully run tasks. And 20000 limit is finished pretty fast for us.Anna Geller
Kevin Kho