Hi folks, we spotted a change in `0.14.11` that made the Flow object no longer pickleable using `pic...

Marwan Sarieddine

03/04/2021, 4:53 PM

Hi folks, we spotted a change in

0.14.11

that made the Flow object no longer pickleable using

pickle

- I am wondering if others have bumped into this

Marwan Sarieddine

03/04/2021, 4:53 PM

Specifically it is due to the use of

lambda

Flow.___init___

self.__slug__counters

which raises this error

Copy code

AttributeError: Can't pickle local object 'Flow.__init__.<locals>.<lambda>'

Marwan Sarieddine

03/04/2021, 4:54 PM

i.e. it is this code block in

Flow.___init___

(

prefect/core/flow.py

)

Copy code

self._slug_counters = collections.defaultdict(
            lambda: itertools.count(1)
        )  # type: Dict[str, Iterator[int]]

Marwan Sarieddine

03/04/2021, 4:54 PM

changing this to use a named function addresses the pickling issue

Marwan Sarieddine

03/04/2021, 4:55 PM

i.e

Copy code

def count():
    return itertools.count(1)

self._slug_counters = collections.defaultdict(count)

Marwan Sarieddine

03/04/2021, 4:55 PM

happy to open a PR with this minor fix if need be

Samuel Hinton

03/04/2021, 4:59 PM

Are you using pickle or cloudpickle?

Marwan Sarieddine

03/04/2021, 5:01 PM

pickle

- sadly we are constrained to using pickle since we are using a prefect flow as part of a tensorflow 2.0 training

Samuel Hinton

03/04/2021, 5:02 PM

Hmm, in that case the easy solution disappears, and I hope a dev can offer some proper insight soon!

Marwan Sarieddine

03/04/2021, 5:24 PM

so as an update - we can circumvent this issue by subclassing

Flow

and adjusting the

subclassed.__init__

accordingly

Jim Crist-Harif

03/04/2021, 5:25 PM

Hi Marwan, we don't guarantee flows are pickleable (only cloudpickleable). Many tasks in prefect are only serializable via cloudpickle (e.g. any

@task

created task), so in practice pickle-only support is not a priority for us. That said, we'd accept a simple fix if you want. I'd use a

partial

here instead to avoid defining a top-level function.

Marwan Sarieddine

03/04/2021, 5:34 PM

Hi @Jim Crist-Harif - thanks for responding, that’s understandable … Sure will open a PR for this simple fix

Marwan Sarieddine

03/04/2021, 6:40 PM

Please see the PR here https://github.com/PrefectHQ/prefect/pull/4209

4 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.