Hi folks - is there an easy way to make a flow pickleable by pickle not cloudpickle ? the reason I am asking is I'd like to use a prefect flow in tensorflow training and tensorflow relies on python's multiprocessing library which uses vanilla pickle

Hi @Marwan Sarieddine! I’m not 100% sure so let me check in with the team and get back to you

thank you

Hi Marwan - are you finding that your Flows are not pickleable by vanilla pickle? If so, would you mind providing the tracebacks you are seeing?

Hi @Chris White - thank you for following up below is the sample script I am using to test things

from prefect import task, Flow
import cloudpickle
import pickle


@task
def func(x):
    return 2 * x

with Flow("test") as flow:
    y = func(2)

cloudpickle.dumps(flow)
pickle.dumps(flow)

and the last line

picke.dumps(flow)

throws this error:

Traceback (most recent call last):
  File "simple_pickle.py", line 14, in <module>
    pickle.dumps(flow)
_pickle.PicklingError: Can't pickle <function func at 0x106f58160>: it's not the same object as __main__.func

I am using prefect version

0.13.4

Ah yes yes; so this is one of the big reasons that

pickle

is not a viable serializer for Prefect workflows - it is very sensitive to import paths and where things are defined. Specifically in this instance,

func

is both the name of the underlying function (defined in

__main__

) as well as the name of the task instance. To be precise:

@task
def func(x):
    return 2 * x

func.__dict__['run'] # <function __main__.func(x)>
func # <Task: func>

pickle.dumps(func) # pickling error
pickle.dumps(func.__dict__['run']) # pickling error

and pickle gets confused about which

func

is which. In theory, a class-based implementation resolves this:

from prefect import Task

class Func(Task):
    def run(self, x):
        return 2 * x

my_func = Func()
pickle.dumps(my_func) # succeeds!

but I suspect you will run into more issues like this the deeper you dig here

I see - thanks for explaining the naming collision - wasn't totally clear to me

Glad to help 👍