Hey im trying to run a multiprocessor inside a prefect task

Question

Hey im trying to run a multiprocessor inside a prefect task function that makes a call to a function not defined as task outside the task It runs fine on my local machine but gives PicklingError when running on local agent refer image

Dylan · Answer

Hey < Anish Chhaparwal> Are you using an executor when you re running with your agent

Anish Chhaparwal · Answer

nope

Dylan · Answer

What s in `query result list`

Dylan · Answer

Also you d define an executor in your `Flow` definition something like ```with Flow name= My Flow environment=LocalEnvironment executor=DaskExecutor as flow flow things```

Dylan · Answer

Executors aren t used when calling `flow run ` by default but are used when a flow run is started by an agent if you ve configured your flow as above point up

Anish Chhaparwal · Answer

hey im a little confused I did not run the flow from the agent I registered it and then ran using the UI I have other flows running on the same agent that work fine

Anish Chhaparwal · Answer

query result list gets a list of documents from database in the form of list It has a few million record so i d like to use multiprocessor

Dylan · Answer

Dylan · Answer

My suggestion is to make `dcmpath check` a task and use Prefect s built in mappping

Dylan · Answer

You can then achieve multiprocessing with the DaskExecutor <https docs prefect io core advanced tutorials dask cluster html the dask executor>

Dylan · Answer

That way your whole flow is executed using multiprocessing

Dylan · Answer

I m not sure why you re experiencing this specific error but I believe that the above suggestion should serve

Dylan · Answer

Dylan · Answer

Note that you don t need to pass an address to the executor if you omit it one will be spun up for you

Anish Chhaparwal · Answer

the reason i didnt use the map was because i d have to fire a dask agent everytime please correct me if im wrong and if i dont pass an address how many workers are spun by default

Dylan · Answer

That s configurable I believe you can pass `n proc` or `n thread` depending on how you re running your scheduler

Dylan · Answer

I m not sure what the default is

Dylan · Answer

To get started you might even try <https docs prefect io api latest engine executors html localdaskexecutor>

Anish Chhaparwal · Answer

okay ill give this a shot and get back to you Meanwhile if you can find out what i did wrong in the first place while using pythons multiprocessor please let me know

Dylan · Answer

I m not sure if you did anything wrong Haha

Dylan · Answer

Looking at your code it feels like this should work

Dylan · Answer

Functions can be pickled

Anish Chhaparwal · Answer

< Dylan> Just ran the program using map task along with dask executor It is significantly slower than just using the python multiprocessor when using the same number of cores Can you look into the original issue again i dont understand why anything has to be pickled i did another trial defining the dcmpath check function function named as dc when running the below trial within the query result list function itself I get the error shown in image

Dylan · Answer

< Anish Chhaparwal> can you share a reproducible example

Dylan · Answer

Prefect is set up for use in distributed environments

Dylan · Answer

So anything that needs to be able to be passed between many machines needs to be serializable

Dylan · Answer

We use cloudpickle for this purpose

Anish Chhaparwal · Answer

okay

Dylan · Answer

I m also confused by your example though because the multiprocessing you re attempting shouldn t leave a single task

Dylan · Answer

So in theory it should work

Dylan · Answer

But if a task doesn t have access to that global function it might be problematic

Dylan · Answer

Actually you could try defining that function within the task method

Anish Chhaparwal · Answer

I did exactly that and still getting an error please refer my previous comment before okay

Dylan · Answer

Can you share a reproducible example as a code snippet

Dylan · Answer

I don t have a complete look at your code from the pictures so far and I d like something I can copy paste smile

Dylan · Answer

Also here s some background that may be helpful <https docs prefect io orchestration faq dataflow html>

Dylan · Answer

ahh < Anish Chhaparwal>

Dylan · Answer

tl dr multiprocessing uses the standard library version of pickle which doesn t work well with functions defined in ` main ` this is one of the issues cloudpickle solves for

Anish Chhaparwal · Answer

cool Thanks for citing the issue link Will import the function as part of lib and try again

Anish Chhaparwal · Answer

shall keep you posted

Anish Chhaparwal · Answer

hey it worked when i imported the function thanks

Dylan · Answer

Awesome Glad to hear it =