Hey anybody got any way to wait on multiple flows at the same time that are created by `create_flow_...

Joshua Greenhalgh

07/29/2022, 1:04 PM

Hey anybody got any way to wait on multiple flows at the same time that are created by

create_flow_run

? Want to do something like;

Joshua Greenhalgh

07/29/2022, 1:05 PM

Copy code

with Flow('backfill_boot') as backfill_boot:
    
    flow_runs = []
    for param in params:
        
        flow_run = create_flow_run(
            flow_name='flow_name',
            project_name='production',parameters=param
        )
        flow_runs.append(flow_run)

    wait_for(flow_runs)

Joshua Greenhalgh

07/29/2022, 1:06 PM

Could probably create something on top of the

wait_for_flow_run

task but was thinking maybe something existed already?

Joshua Greenhalgh

07/29/2022, 1:10 PM

This kinda works;

Copy code

for flow_run in flow_runs:
        res = wait_for_flow_run(flow_run)

But each call to wait_for_flow_run blocks - would be nice to be able to stream the logs interleaved for all the flow_runs I am waiting for!

James Sopkin

07/29/2022, 1:50 PM

Hi Joshua, by default, prefect 1.0 runs sequentially. You could try adding an executor if you want parallelism

James Sopkin

07/29/2022, 1:52 PM

so for a code example-

Copy code

with Flow('backfill_boot', executor=LocalDaskExecutor()) as backfill_boot:

Joshua Greenhalgh

07/29/2022, 3:17 PM

Thanks - I don't think it really needs task parallelism to accomplish - could have a single task that takes the ids you care about and within the task loop through and wait for each?

Joshua Greenhalgh

07/29/2022, 3:17 PM

I can probably write it myself though!

James Sopkin

07/29/2022, 3:37 PM

ahh, in that case a for loop like the one you had should be fine then

4 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.