Hi there If I pass an object to a task (with the dask runtime). What exactly will happen to this obj...

Michael Hadorn

02/01/2021, 8:25 AM

Hi there If I pass an object to a task (with the dask runtime). What exactly will happen to this object? I was thinking, it would be serialized and copied for every task. But it seems, that there will also be some logic runned. Is this process documented anywhere? Sure I can give more details, if needed.

✅ 1

Michael Hadorn

02/01/2021, 9:48 AM

Ok to specify my question. how is cloudpickle working? My object i wanna process with tasks need some data from the database first. In the task the object can be used without database, because all the needed properties are correctly filled. Currently every worker is redo the database initialized, although this is not in the init method.

Dylan

02/01/2021, 5:53 PM

Hi @Michael Hadorn! I’m not sure that I totally understand your question. Could you provide a short code snippet to illustrate the behavior that’s confusing?

Michael Hadorn

02/03/2021, 8:29 AM

Hi Dylan Thx for your answer. Sorry didn't got the notification. My problem is solved now. I messed up the init part of my flow, thats way some code was run again on every dask worker. Best wishes Michael

Dylan

02/03/2021, 3:39 PM

Glad you got it sorted!

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.