https://prefect.io logo
j

Jacques

10/06/2020, 2:04 PM
We used to use
LocalDaskExecutor(scheduler="threads")
for running one-off flows, then started having issues with tasks being repeated and was recommended we switch to
DaskExecutor
, which we've been using with
DaskExecutor(cluster_kwargs = {"processes": False})
. We've now run into some new bugs with latest prefect (a side note, but here: https://github.com/PrefectHQ/prefect/issues/3443) - trying to understand what the difference is between
LocalDaskExecutor
and
DaskExecutor
using threads and a temporary cluster? Is there a performance/reliability advantage of using one over the other?
j

Jim Crist-Harif

10/06/2020, 2:14 PM
Hi @Jacques, • The
LocalDaskExecutor
should no longer ever repeat tasks, that was fixed in 0.13.2 • If you're using
processes=False
, there should be no difference in performance between
LocalDaskExecutor()
and
DaskExecutor
. In cases like that I'd recommend the former as its lighter weight. Either should work fine though. • Thanks for filing the bug report, I'm sorry you're running into issues. Taking a look now.
j

Jacques

10/06/2020, 2:17 PM
Awesome, thanks so much for the quick response - we'll switch to using LocalDaskExecutor then!