I have a last question and then I think I'm good to go. I have a local worker that calls code that uses multiprocessing. Daemonized processes aren't allowed to spawn children, so my code obviously fails. Is it possible to run code as a subprocess that is not daemonized, some how?
k
Kevin Kho
09/14/2021, 2:01 PM
Hey @Mikkel Antonsen, I do not believe so. Really, the easiest way do parallel process is to use the LocalDaskExecutor. If you are using either LocalDaskExecutor or DaskExecutor, think of it that all the cores or compute are already utilized to there is no room for a second stage multiprocessing.
m
Mikkel Antonsen
09/15/2021, 11:05 AM
The multiprocessing is done by a library I'm using so it's "Outside" of my control
k
Kevin Kho
09/15/2021, 1:54 PM
I think you’re limited to using the LocalExecutor then because Dask will really stop any multiprocessing code from running and raise the error
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.