Peter Roelants
05/21/2021, 1:20 PMt
minutes
◦ Calls flow_B x
times using StartFlowRun
◦ Run with LocalDaskExecutor
to limit parallelism to max y
runs at a time. For one scheduled run, the dependent flow flow_B is limited to run max y
at a time.
Now, I noticed that for a single run of flow_A, flow_B is indeed limited to a parallelism of y
runs at one time.
However, when a previous flow_A (and dependent *flow_B*s) are still running), and a new flow_A, with new dependent *flow_B*s are scheduled than more than y
*flow_B*s can run at the same time.
For example with parallelism `y=2`:
• At time 1 there will be 2 *flow_B*s runing.
Run flow_A 1:
|--dependent flow_B 1: running
|--dependent flow_B 2: running
|--dependent flow_B 3: waiting
• At time 2 there will be 4 *flow_B*s runing.
Running flow_A-1:
|--dependent flow_B-1: finished
|--dependent flow_B-2: running
|--dependent flow_B-3: running
Run flow_A-2:
|--dependent flow_B-4: running
|--dependent flow_B-5: running
|--dependent flow_B-6: waiting
Is it possible to limit the parallelism of flow_B to max y
over all scheduled runs?Kevin Kho
05/21/2021, 1:27 PMPeter Roelants
05/21/2021, 1:45 PMKevin Kho
05/21/2021, 1:47 PMPeter Roelants
05/21/2021, 5:56 PMKevin Kho
05/21/2021, 6:31 PM