Luis Gallegos
01/20/2021, 6:22 PMRuslan
01/20/2021, 6:33 PMJim Crist-Harif
01/20/2021, 6:39 PMStartFlowRun
and use a single "orchestrating" flow to run each flow in turn. If the orchestrating flow is run with a LocalDaskExecutor
, only num_workers
flows will be executed at a time. For example:
start_flow_run = StartFlowRun("your-csv-flow", wait=True)
with Flow("orchestrating-flow") as flow:
# run_parameters takes a list of dicts of parameters to run your-csv-flow with
run_parameters = Parameter("run_parameters")
start_flow_run.map(parameters=run_parameters)
flow.executor = LocalDaskExecutor(num_workers=8) # only 8 flows will run concurrently
See https://docs.prefect.io/core/idioms/flow-to-flow.html#scheduling-a-flow-of-flows for more information on using StartFlowRun
.Jim Crist-Harif
01/20/2021, 6:40 PMDaskExecutor
or LocalDaskExecutor
) would handle manage your resources, ensuring that tasks only run when resources are free. Either pattern works, depends on how you intend to use these flows.Luis Gallegos
01/20/2021, 7:01 PMRuslan
01/20/2021, 7:04 PMLuis Gallegos
01/20/2021, 7:09 PM