https://prefect.io logo
Title
t

trapped

03/02/2020, 8:09 AM
hi guys! is there any way to limit flow run (not task run) concurrency for agents?
a

Alex Goodman

03/02/2020, 1:03 PM
There are no built in mechanisms to limit concurrency at a flow-run level in the agent at this time. If you are using Prefect Cloud, the closest feature is configuring task tag concurrency (https://docs.prefect.io/cloud/concepts/task-concurrency-limiting.html) --which I realize is not at a flow-run level like you are asking for. You could create your own Agent class that inherits from the DockerAgent (or another agent class of choice) to get the desired effect. Specifically, to implement flow-run throttling at the agent (by limiting the result of
query_flow_runs
https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/agent.py#L313), however, this would not limit concurrency, it would only serve to throttle how often flow runs are started. To limit concurrency you would need to monitor the number of running flow runs to influence the result of
query_flow_runs
. Assuming you are using the DockerAgent, this would mean monitoring the number of child processes (which is roughly equal to the number of flow runs be run by this agent): https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/docker/agent.py#L87
t

trapped

03/02/2020, 2:00 PM
that's perfect! thanks
hi! just a quick followup; I ended up implementing it as you suggested, subclassing the
Agent
class and redefining the
agent_process
method; works great!