Alexander10/23/2020, 7:54 PM
(i use local one).
Using executor type DaskExecutor
I expect leaf1 and leaf2 to run in parallel but they are not. In gantt chart i see they running sequentially. If i run them locally, they run in parallel. BTW whats the difference between environment executor and flow executor? Flows are run by docker agent.
import time from prefect import task, Flow from prefect.engine.executors import DaskExecutor from prefect.environments import LocalEnvironment @task def wait_for(seconds): time.sleep(seconds) with Flow('_Concurrency_') as flow: head = wait_for(5) leaf1 = wait_for(30) leaf2 = wait_for(15) tail = wait_for(3) head.set_dependencies(downstream_tasks=[leaf1, leaf2]) leaf1.set_downstream(tail) leaf2.set_downstream(tail) flow.environment = LocalEnvironment(executor=DaskExecutor()) flow.executor = DaskExecutor()
from prefect.engine.executors import LocalDaskExecutor ### flow.environment = LocalEnvironment(executor=LocalDaskExecutor(scheduler="threads", num_workers=6))
) and the environment executor is used when running them with an environment.
Alexander10/23/2020, 8:10 PM
num_workers = dask.config.get("num_workers", None) or CPU_COUNT
ale10/24/2020, 6:18 AM
, you can read about that here, with the primary difference between the two being the scheduler; the
uses the Dask distributed scheduler while the
can use any scheduler but doesn't leverage all the distributed features of Dask.