Jacques
10/06/2020, 2:04 PMLocalDaskExecutor(scheduler="threads")
for running one-off flows, then started having issues with tasks being repeated and was recommended we switch to DaskExecutor
, which we've been using with DaskExecutor(cluster_kwargs = {"processes": False})
. We've now run into some new bugs with latest prefect (a side note, but here: https://github.com/PrefectHQ/prefect/issues/3443) - trying to understand what the difference is between LocalDaskExecutor
and DaskExecutor
using threads and a temporary cluster? Is there a performance/reliability advantage of using one over the other?Jim Crist-Harif
10/06/2020, 2:14 PMLocalDaskExecutor
should no longer ever repeat tasks, that was fixed in 0.13.2
• If you're using processes=False
, there should be no difference in performance between LocalDaskExecutor()
and DaskExecutor
. In cases like that I'd recommend the former as its lighter weight. Either should work fine though.
• Thanks for filing the bug report, I'm sorry you're running into issues. Taking a look now.Jacques
10/06/2020, 2:17 PM