Hi Team, I have Prefect Cloud setup on GKE cluster with Dask also which is being used as the Executor. Can I use

multiprocessing

inside my task code and how is that expected to behave in Prefect+Dask? Code looks like this inside the task:

from tqdm import tqdm
import multiprocessing

pool = multiprocessing.Pool(processes=16)
func_output = list(tqdm(pool.imap(some_func, some_functions_input)))

I am getting this error:

AssertionError: daemonic processes are not allowed to have children

Hey @Nivi Mukka, I believe you can use multiprocessing. Someone a few threads above did it. I think it it’s a bit tricky. If you use the DaskExecutor and then multiprocess inside, it would be mapped to the workers and then run inside the workers, but I’m honestly not sure there is a guarantee that you end up cleanly with one mapped task per worker.

I’m not able to use

multiprocessing

- I keep getting the

AssertionError: daemonic processes are not allowed to have children

.

It needs to be split out into it’s own task, have you seen the example in the docs ? There was someone here who said they used multiprocessing so I thought you could, but I suppose it wasn’t on Dask then.

I was able to do the mapping at the flow level and each iteration of the loop is running in parallel but it is still using only one Dask Worker.

Is your flow code simple enough to share?

This is with

flow.register

. Flow code is hard to share.

Could you DM me just the Flow block?