I'm encountering a persistent issue with Dask serialization when updating my flows from Prefect
2.19.1
to Prefect
3.1.2
. Every run incurs the following error, which I've never seen before in 18 months of using Prefect 2:
Task run failed with exception: TypeError('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 1 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x7f4e3b547400>\n 0. 139974151916736\n>') - Retries are exhausted
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 63, in dumps
result = pickle.dumps(x, **dump_kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/docker/__init__.py", line 20, in __getattr__
raise ImportError(f"module {__name__!r} has no attribute {name!r}")
ImportError: module 'prefect.docker' has no attribute 'isnan'
<---Long series of stack traces--->
The above exception was the direct cause of the following exception:
<<---Long stack trace--->>
File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/serialize.py", line 392, in serialize
raise TypeError(msg, str_x) from exc
TypeError: ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 1 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x7f4e3b547400>\n 0. 139974151916736\n>')
All of my flows write to Zarr N-Dimensional data stores using Xarray's
to_zarr
method,.
to_zarr
leverages Dask under the hood to vastly speed up this operation. I don't use Dask Task Runners or any specialized Prefect Dask objects.
Digging around I've found this conversation, which doesn't seem directly related
https://linen.prefect.io/t/23198687/hello-everyone-i-m-facing-an-issue-using-prefect-with-a-dask. Any ideas Prefect team?
I'll post a full stack trace in follow up