Multiprocessing library does not work inside a prefect flow? My code works fine outside of prefect, but breaks when inside prefect. I create a mp pool like such:

mp.Pool(n_processes if n_processes <= os.cpu_count() else os.cpu_count())

and run my code. This breaks when inside. a prefect flow. If I set

n_processes

to 1 then the flow succeeds. I think I'm encountering some deadlock situation?

I had similar problems but was able to get multiprocessing working using a

ThreadPool

this doesn't actually use more cores though. I'm looking into dask

I honestly prefer dask if running more than one function in parallel. It’s a smoother experience, albeit with more overhead