Avi A
05/30/2020, 10:00 PMLocalDaskExecutor
. I keep getting the following error messages, which are probably related:
Error message: can't start new thread
Error message: 'DummyProcess' object has no attribute 'terminate'
BlockingIOError: [Errno 11] Resource temporarily unavailable
Alex Cano
05/30/2020, 10:15 PMDaskExecutor
? Without much more detail on where those error messages come from, it might be a bit tough to debug (without someone intimately familiar w/ Dask)Avi A
05/31/2020, 8:31 AMFlow
to use it, like so:
with Flow() as flow:
...
flow.environment = RemoteEnvironment(
executor="prefect.engine.executors.LocalDaskExecutor",
executor_kwargs={'scheduler': 'threads'},
)
flow.register()
I’m seeing the error in the UI when I try to run it.
I haven’t tried using DaskExecutor
, I see no reason to since I’m only using one machineAlex Cano
05/31/2020, 8:04 PMDaskExecutor
, since the core underlying differences between the DaskExecutor
and the LocalDaskExecutor
is which scheduler is used under the hood. One thing I will note is that I’m not aware of a scenario where using the LocalDaskExecutor
is preferable to the DaskExecutor
itself, so that’s probably a place for someone more familiar with Dask to chime in. See below for the link to the docs on Dask itself and the differing kinds of schedulers.
Different schedulers link: https://docs.dask.org/en/latest/scheduling.html
Some more docs on using the distributed scheduler (DaskExecutor
) on a single machine: https://docs.dask.org/en/latest/setup/single-distributed.html
Hopefully these links can help!