Hi folks, we spotted a change in `0.14.11` that ma...
# ask-communitym
Marwan Sarieddine
03/04/2021, 4:53 PMHi folks, we spotted a change in
0.14.11pickleMarwan Sarieddine
03/04/2021, 4:53 PMSpecifically it is due to the use of
lambdaFlow.___init___self.__slug__counters Copy code
AttributeError: Can't pickle local object 'Flow.__init__.<locals>.<lambda>'Marwan Sarieddine
03/04/2021, 4:54 PMi.e. it is this code block in
Flow.___init___prefect/core/flow.py Copy code
self._slug_counters = collections.defaultdict(
lambda: itertools.count(1)
) # type: Dict[str, Iterator[int]]Marwan Sarieddine
03/04/2021, 4:54 PMchanging this to use a named function addresses the pickling issue
Marwan Sarieddine
03/04/2021, 4:55 PMi.e
Copy code
def count():
return itertools.count(1)
self._slug_counters = collections.defaultdict(count)Marwan Sarieddine
03/04/2021, 4:55 PMhappy to open a PR with this minor fix if need be
s
Samuel Hinton
03/04/2021, 4:59 PMAre you using pickle or cloudpickle?
m
Marwan Sarieddine
03/04/2021, 5:01 PMpickles
Samuel Hinton
03/04/2021, 5:02 PMHmm, in that case the easy solution disappears, and I hope a dev can offer some proper insight soon!
m
Marwan Sarieddine
03/04/2021, 5:24 PMso as an update - we can circumvent this issue by subclassing
Flowsubclassed.__init__j
Jim Crist-Harif
03/04/2021, 5:25 PMHi Marwan, we don't guarantee flows are pickleable (only cloudpickleable). Many tasks in prefect are only serializable via cloudpickle (e.g. any
@taskpartialm
Marwan Sarieddine
03/04/2021, 5:34 PMHi @Jim Crist-Harif - thanks for responding, that’s understandable … Sure will open a PR for this simple fix
Marwan Sarieddine
03/04/2021, 6:40 PMPlease see the PR here https://github.com/PrefectHQ/prefect/pull/4209
2 Views