https://prefect.io logo
c

Cody

08/07/2023, 9:10 PM
Has anyone used
deployments.run_deployment
with
timeout=0
as a way to break a single stream of flows into two or more parallel streams of totally independent flows without needing to bother with async stuff? I haven't seen it used, and would be interested in hearing if anyone sees any badness coming from this pattern. There's an example that I just ran in the thread here.
Copy code
from prefect import flow, task, deployments

from flows.module_a.script_a import func_a
from flows.module_b.script_b import func_b
from flows.module_c.script_c import func_c
from flows.module_d.script_d import func_d
from flows.module_e.script_e import func_e


@flow(log_prints=True)
def flow_schedule():
    func_a()
    func_b()
    func_c()
    flow_splitter(flows_to_run=['d', 'e'])


@flow(log_prints=True)
def flow_splitter(flows_to_run):
    """Setting the timeout on a flow=0 will let this function return without waiting for the flow to finish.
    This is useful if you want to run multiple flows in parallel.
    Note - flows run this way will not be aware of each other, so they must be totally independent."""
    flow_dict = {'a': 'func-a/default',
                 'b': 'func-b/default',
                 'c': 'func-c/default',
                 'd': 'func-d/default',
                 'e': 'func-e/default', }
    for f in flows_to_run:
        print(f'Running {f}: {flow_dict[f]}')
        deployments.run_deployment(name=flow_dict[f], timeout=0)


if __name__ == '__main__':
    flow_schedule()
d

Deceivious

08/08/2023, 5:16 AM
Ive used the run deployment and it works.
No downsides so far. 1. Just make sure the agent/workers have the bandwidth to handle the new flow runs, else they go in late state.
c

Cody

08/08/2023, 3:33 PM
For once I'm happy to say that we use AWS ECS lol