Hagai Arad
10/09/2020, 10:31 AMFlowRunTask
and run them concurrently using local dask executor. (The flow code is in the first comment).
My question is, is there another way to make the FlowRunTask
’s run concurrently without using dask? Thanks!Hagai Arad
10/09/2020, 10:32 AMa_flow = FlowRunTask(flow_name='a flow', project_name='test', wait=True, name='a flow - sub_flow_of_abcd')
b_flow = FlowRunTask(flow_name='b flow', project_name='test', wait=True, name='b flow - sub_flow_of_abcd')
c_flow = FlowRunTask(flow_name='c flow', project_name='test', wait=True, name='c flow - sub_flow_of_abcd')
d_flow = FlowRunTask(flow_name='d flow', project_name='test', wait=True, name='d flow - sub_flow_of_abcd')
with Flow(name="abcd scenario test",
environment=LocalEnvironment(executor=DaskExecutor()),
storage=Docker(base_image='prefect_image',
image_tag='latest',
local_image=True)) as flow:
c = c_flow(upstream_tasks=[a_flow, b_flow])
d = d_flow(upstream_tasks=[a_flow])
nicholas
Jim Crist-Harif
10/09/2020, 3:43 PMLocalDaskExecutor()
instead, as its much lighter weight.Jim Crist-Harif
10/09/2020, 3:43 PMDaskExecutor
requires.