Alexander
10/23/2020, 7:54 PMUsing executor type DaskExecutor
(i use local one).
import time
from prefect import task, Flow
from prefect.engine.executors import DaskExecutor
from prefect.environments import LocalEnvironment
@task
def wait_for(seconds):
time.sleep(seconds)
with Flow('_Concurrency_') as flow:
head = wait_for(5)
leaf1 = wait_for(30)
leaf2 = wait_for(15)
tail = wait_for(3)
head.set_dependencies(downstream_tasks=[leaf1, leaf2])
leaf1.set_downstream(tail)
leaf2.set_downstream(tail)
flow.environment = LocalEnvironment(executor=DaskExecutor())
flow.executor = DaskExecutor()
I expect leaf1 and leaf2 to run in parallel but they are not. In gantt chart i see they running sequentially. If i run them locally, they run in parallel.
BTW whats the difference between environment executor and flow executor?
Flows are run by docker agent.nicholas
from prefect.engine.executors import LocalDaskExecutor
###
flow.environment = LocalEnvironment(executor=LocalDaskExecutor(scheduler="threads", num_workers=6))
flow.run
) and the environment executor is used when running them with an environment.Alexander
10/23/2020, 8:10 PMnum_workers = dask.config.get("num_workers", None) or CPU_COUNT
🤦♂️ale
10/24/2020, 6:18 AMnicholas
LocalDaskExecutor
and DaskExecutor
, you can read about that here, with the primary difference between the two being the scheduler; the DaskExecutor
uses the Dask distributed scheduler while the LocalDaskExecutor
can use any scheduler but doesn't leverage all the distributed features of Dask.