Ievgenii Martynenko
03/24/2022, 9:44 AM# with concurrent.futures.thread.ThreadPoolExecutor(max_workers=10,
# thread_name_prefix="flow_") as executor:
# futures = [executor.submit(self._run, df) for df in self.dfs]
#
# for future in as_completed(futures):
# try:
# future.result()
# except Exception as exc:
# logger.exception(exc)
Anna Geller
03/24/2022, 1:34 PMLocalDaskExecutor
that uses multithreading under the hood by default, then Prefect will make sure that the context stays thread safeIevgenii Martynenko
03/24/2022, 2:23 PMKevin Kho
03/24/2022, 2:46 PMlogger
as an input into your multiprocessing function, it will work, but really just use LocalDask and then you get observability into your tasks as wellAnna Geller
03/24/2022, 3:34 PMYeap, for DaskCluster its clear. It controls multithreading itself and that all things work fine.Actually, using
LocalDaskExecutor
doesn't use Dask, it uses multithreading. Check out https://discourse.prefect.io/t/what-is-the-difference-between-a-daskexecutor-and-a-localdaskexecutor/374