https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jon Ruhnke

08/11/2022, 7:11 PM
In Prefect 1.0... How do I run multiple instances of the same flow, based on a list of data? An example would be I've created a generic flow to go grab 1 table from a source system. I want to be able to neatly specify dozens of tables in a convenient location (config file?) and have an instance of the flow run for each of the tables. Do I need to create a 'master' flow that loops through and runs a bunch of child flows? I've already created a working flow with the .map() function to iterate over the list of tables, but that doesn't seem like a good approach since the Prefect UI doesn't seem to neatly separate things out for debugging and retrying.
n

Nate

08/11/2022, 7:22 PM
you could use
create_flow_run.map([{'child_arg': 'value'}, ...], flow_name=unmapped(my_flow_name))
or you could define your own task using the python client, which you could again map over like
Copy code
@task
def trigger_flow_run(arg: Tuple) -> None:
    child_flow_name, params = arg
    print(f'Starting child flow run {child_flow_name}')
    client = Client()
    client.create_flow_run(
        run_name=f"{context.flow_run_name}-{child_flow_name}",
        version_group_id="YOUR_CHILD_FLOWS_VERSION_GROUP_ID",
        parameters=params

with Flow('Parent Flow') as flow:
    children = [
        ('child_flow_1', {'child_arg': 'value_1'}), ('child_flow_2', {'child_arg': 'value_2'})
    ]

    trigger_flow_run.map(children)
j

Jon Ruhnke

08/11/2022, 7:53 PM
I see create_flow_run requires a registered flow and prefect agent. How would I go about testing this during development on local CLI?
with Flow('pipeline-sap-child') as child_flow:
entity_config = assemble_config({'name': 't141t'}, 'sap')
raw_data = extract(entity_config)
with Flow('pipeline-sap-parent') as parent_flow:
child_run_id = create_flow_run(flow_name='pipeline-sap-child')
if __name__ == '__main__':
parent_flow.run()
a

Anna Geller

08/12/2022, 1:49 PM
not possible - create_flow_run is orchestrator pattern which means it needs access to the backend API and can't run locally without prior registration of child and parent flows
check resources here to learn more https://discourse.prefect.io/tag/flow-of-flows
5 Views