Hi, I have a flow that uses several `FlowRunTask` ...
# prefect-server
h
Hi, I have a flow that uses several
FlowRunTask
and run them concurrently using local dask executor. (The flow code is in the first comment). My question is, is there another way to make the
FlowRunTask
’s run concurrently without using dask? Thanks!
Copy code
a_flow = FlowRunTask(flow_name='a flow', project_name='test', wait=True, name='a flow - sub_flow_of_abcd')
b_flow = FlowRunTask(flow_name='b flow', project_name='test', wait=True, name='b flow - sub_flow_of_abcd')
c_flow = FlowRunTask(flow_name='c flow', project_name='test', wait=True, name='c flow - sub_flow_of_abcd')
d_flow = FlowRunTask(flow_name='d flow', project_name='test', wait=True, name='d flow - sub_flow_of_abcd')

with Flow(name="abcd scenario test",
          environment=LocalEnvironment(executor=DaskExecutor()),
          storage=Docker(base_image='prefect_image',
                         image_tag='latest',
                         local_image=True)) as flow:
    c = c_flow(upstream_tasks=[a_flow, b_flow])
    d = d_flow(upstream_tasks=[a_flow])
n
Hi @Hagai Arad - unfortunately not, Dask is the way that Prefect handles task parallelism.
j
For your use case, I recommend using the
LocalDaskExecutor()
instead, as its much lighter weight.
upvote 1
This will run all tasks in the same process using threads, avoiding the extra machinery that the
DaskExecutor
requires.
upvote 1