Cody
08/07/2023, 9:10 PMdeployments.run_deployment
with timeout=0
as a way to break a single stream of flows into two or more parallel streams of totally independent flows without needing to bother with async stuff?
I haven't seen it used, and would be interested in hearing if anyone sees any badness coming from this pattern.
There's an example that I just ran in the thread here.from prefect import flow, task, deployments
from flows.module_a.script_a import func_a
from flows.module_b.script_b import func_b
from flows.module_c.script_c import func_c
from flows.module_d.script_d import func_d
from flows.module_e.script_e import func_e
@flow(log_prints=True)
def flow_schedule():
func_a()
func_b()
func_c()
flow_splitter(flows_to_run=['d', 'e'])
@flow(log_prints=True)
def flow_splitter(flows_to_run):
"""Setting the timeout on a flow=0 will let this function return without waiting for the flow to finish.
This is useful if you want to run multiple flows in parallel.
Note - flows run this way will not be aware of each other, so they must be totally independent."""
flow_dict = {'a': 'func-a/default',
'b': 'func-b/default',
'c': 'func-c/default',
'd': 'func-d/default',
'e': 'func-e/default', }
for f in flows_to_run:
print(f'Running {f}: {flow_dict[f]}')
deployments.run_deployment(name=flow_dict[f], timeout=0)
if __name__ == '__main__':
flow_schedule()
Deceivious
08/08/2023, 5:16 AMCody
08/08/2023, 3:33 PM