Currently, Prefect has no notion of a task queue that would allow load-balancing flow runs across multiple agents based on available resources on each agent. If you need such functionality, you would need to assign unique labels to your agents, and then assign corresponding labels to your flows to spread the load across multiple agents. For instance: Flow 1:
$ prefect agent local start --no-hostname-label --label prod1 $ prefect agent local start --no-hostname-label --label prod2
from prefect.run_configs import UniversalRun ... with Flow(name="example1", run_config=UniversalRun(labels=["prod1"])) as flow:
To help with scale, we recommend using one of the other agents, such as: • KubernetesAgent • ECSAgent • VertexAgent
from prefect.run_configs import UniversalRun ... with Flow(name="example2", run_config=UniversalRun(labels=["prod2"])) as flow:
agent is shut down for some reason the flows scheduled for this
will not be executed at all. How do you handle it? 2. We have mapped tasks in our flows. If mapped task is split to 100 tasks it will be counted as 100 separate successfully run tasks. And 20000 limit is finished pretty fast for us.