Geoff Coyner11/23/2022, 1:31 AM
will create all of flow runs at once, which is clogging up my agent's backlog when N gets too large. This creates a lot of failures, which occur before my child flow runs can even begin. Unfortunately, I am not the admin and don't have control over the agent we're using, so I don't have a way to easily investigate why this is or resolve it. Even if I was, it seems like kicking off this many flows at once probably isn't a good idea. Thus, I'm wondering if there is some way to throttle
my flow without pre-defining my N child flows. I'm even OK with running all N flows in sequence. I.e. kick off flow run 1 -> wait for flow run 1 -> kick off flow run 2 -> etc. Here is the structure I'm using now:
My Question: Is there some way to combine the functionality of`create_flow_run` and
def create_wait_on_child(parameters): create_id = create_flow_run(flow_name="GenericChild", project_name="Project", parameters=parameters) return wait_for_flow_run(create_id, raise_final_state=True) with Flow('Parent Flow') as flow: flow_runs_params_list = get_flow_runs_params() apply_map(schedule_run_backfill, flow_runs_params_list)
in the same task? I know I can't just add a
in the example above (since that would involve tasks with tasks), but that is the functionality I'm going for. Or does anyone have alternative recommendations? Appreciate anyone's input.
wasn't a give-away)
Zanie11/23/2022, 3:46 PM
task combines the creation and waiting operations.
into batches then map over the batches?
Erik Mathis11/23/2022, 3:56 PM
Geoff Coyner11/23/2022, 5:59 PM
to work. Thanks for the suggestion!