Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi there
If I pass an object to a task (with the dask runtime). What exactly will happen to this object? I was thinking, it would be serialized and copied for every task.
But it seems, that there will also be some logic runned.
Is this process documented anywhere?
Sure I can give more details, if needed.

Ok to specify my question. how is cloudpickle working?
My object i wanna process with tasks need some data from the database first.
In the task the object can be used without database, because all the needed properties are correctly filled.
Currently every worker is redo the database initialized, although this is not in the init method.

Hi <@U01F61UTFUH>!

I’m not sure that I totally understand your question. Could you provide a short code snippet to illustrate the behavior that’s confusing?

Hi Dylan
Thx for your answer. Sorry didn't got the notification.
My problem is solved now. I messed up the init part of my flow, thats way some code was run again on every dask worker.
Best wishes
Michael