Matt Denno
02/04/2021, 8:25 PM@task(checkpoint=False)
def fetch() -> ProtoBufObject:
return fetch_proto_buff_object()
In the past I have set checkpoint=False to avoid serialization errors with the ProtoBuff objects, but now, even with checkpoint=False I am getting the following when i try to register the flow.
TypeError: can't pickle google.protobuf.pyext._message.MessageDescriptor objects
To resolve this I tried to create a serializer for the ProtoBuff like so:
class PBSerializer(Serializer):
def serialize(self, value: Any) -> bytes:
# transform a Python object into bytes
return value.SerializeToString()
def deserialize(self, value: bytes) -> Any:
# recover a Python object from bytes
ts = amanzi_pb2.TimeSeries
return ts.ParseFromString(value)
and called like:
@task(checkpoint=False, result=LocalResult(serializer=PBSerializer()))
def fetch() -> ProtoBufObject:
return fetch_proto_buff_object()
But I get the same error still. It doesn't seem like the result serializer is being used, but it is also not causing any errors. I have tried to debug and step through the code but can't figure out what is happening.
Any help, guidance, ideas for how to resolve would be welcomed.
I am on v 0.13.19Zanie
Matt Denno
02/04/2021, 8:55 PMZanie
MessageDescriptor
object somewhere in your flow / task files?Matt Denno
02/04/2021, 9:03 PMFlorian K. (He/Him)
02/04/2021, 11:43 PMMessageDescriptor
object. A quick check shows me that they hold a FileDescriptor
object which itself extends DescriptorBase
. However, I am not delaring a global variable that holds a PB message.
Now that we identified typing as the culprit I am wondering if this is a bug or the result of prefects architecture....Any ideas?
Thanks,Zanie
flow.register(build=False)
?Florian K. (He/Him)
02/05/2021, 12:31 AMZanie
Jim Crist-Harif
02/06/2021, 5:57 AMfrom __future__ import annotations
In python 3.10 this will become the default behavior: https://www.python.org/dev/peps/pep-0563/Matt Denno
02/08/2021, 2:42 PM