Hagai Arad
10/09/2020, 10:31 AMFlowRunTask
and run them concurrently using local dask executor. (The flow code is in the first comment).
My question is, is there another way to make the FlowRunTask
’s run concurrently without using dask? Thanks!a_flow = FlowRunTask(flow_name='a flow', project_name='test', wait=True, name='a flow - sub_flow_of_abcd')
b_flow = FlowRunTask(flow_name='b flow', project_name='test', wait=True, name='b flow - sub_flow_of_abcd')
c_flow = FlowRunTask(flow_name='c flow', project_name='test', wait=True, name='c flow - sub_flow_of_abcd')
d_flow = FlowRunTask(flow_name='d flow', project_name='test', wait=True, name='d flow - sub_flow_of_abcd')
with Flow(name="abcd scenario test",
environment=LocalEnvironment(executor=DaskExecutor()),
storage=Docker(base_image='prefect_image',
image_tag='latest',
local_image=True)) as flow:
c = c_flow(upstream_tasks=[a_flow, b_flow])
d = d_flow(upstream_tasks=[a_flow])
nicholas
Jim Crist-Harif
10/09/2020, 3:43 PMLocalDaskExecutor()
instead, as its much lighter weight.DaskExecutor
requires.