https://prefect.io logo
Title
h

Hagai Arad

10/09/2020, 10:31 AM
Hi, I have a flow that uses several
FlowRunTask
and run them concurrently using local dask executor. (The flow code is in the first comment). My question is, is there another way to make the
FlowRunTask
’s run concurrently without using dask? Thanks!
a_flow = FlowRunTask(flow_name='a flow', project_name='test', wait=True, name='a flow - sub_flow_of_abcd')
b_flow = FlowRunTask(flow_name='b flow', project_name='test', wait=True, name='b flow - sub_flow_of_abcd')
c_flow = FlowRunTask(flow_name='c flow', project_name='test', wait=True, name='c flow - sub_flow_of_abcd')
d_flow = FlowRunTask(flow_name='d flow', project_name='test', wait=True, name='d flow - sub_flow_of_abcd')

with Flow(name="abcd scenario test",
          environment=LocalEnvironment(executor=DaskExecutor()),
          storage=Docker(base_image='prefect_image',
                         image_tag='latest',
                         local_image=True)) as flow:
    c = c_flow(upstream_tasks=[a_flow, b_flow])
    d = d_flow(upstream_tasks=[a_flow])
n

nicholas

10/09/2020, 3:42 PM
Hi @Hagai Arad - unfortunately not, Dask is the way that Prefect handles task parallelism.
j

Jim Crist-Harif

10/09/2020, 3:43 PM
For your use case, I recommend using the
LocalDaskExecutor()
instead, as its much lighter weight.
:upvote: 1
This will run all tasks in the same process using threads, avoiding the extra machinery that the
DaskExecutor
requires.
:upvote: 1