Peter Roelants

    Peter Roelants

    1 year ago
    Hi Prefect, Is there a way to limit parallel flows in Prefect Server? For example I have two flows: • flow_B Resource intensive, long-running flow that is run as a dependent flow for flow_A flow_A : ◦ Flow that is scheduled to run every
    t
    minutes ◦ Calls flow_B
    x
    times using
    StartFlowRun
    ◦ Run with
    LocalDaskExecutor
    to limit parallelism to max
    y
    runs at a time. For one scheduled run, the dependent flow flow_B is limited to run max
    y
    at a time. Now, I noticed that for a single run of flow_A, flow_B is indeed limited to a parallelism of
    y
    runs at one time. However, when a previous flow_A (and dependent flow_Bs) are still running), and a new flow_A, with new dependent flow_Bs are scheduled than more than
    y
    flow_Bs can run at the same time. For example with parallelism y=2: • At time 1 there will be 2 flow_Bs runing.
    Run flow_A 1:
      |--dependent flow_B 1: running
      |--dependent flow_B 2: running
      |--dependent flow_B 3: waiting
    • At time 2 there will be 4 flow_Bs runing.
    Running flow_A-1:
      |--dependent flow_B-1: finished
      |--dependent flow_B-2: running
      |--dependent flow_B-3: running
    
    Run flow_A-2:
      |--dependent flow_B-4: running
      |--dependent flow_B-5: running
      |--dependent flow_B-6: waiting
    Is it possible to limit the parallelism of flow_B to max
    y
    over all scheduled runs?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Peter Roelants, there is flow concurrency limiting, but only on cloud unfortunately. I don’t there’s a way around this in this specific scenario.
    Peter Roelants

    Peter Roelants

    1 year ago
    Thanks for the reply. Is there any plan to add this feature to Prefect server in the future?
    Kevin Kho

    Kevin Kho

    1 year ago
    Unfortunately, no. Check this
    Peter Roelants

    Peter Roelants

    1 year ago
    The newly suggested fix (to limit runs per agent) as mentioned at the end would probably solve my use case though: https://github.com/PrefectHQ/server/issues/213 Thanks for the reference 🙂
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes feel free to chime in on that issue.