Hi all — I’m totally new to Prefect and Dask, but ran into an issue today that burned a few hours and wanted to make others aware. I inadvertantly upgraded the dependency

msgpack

to

v1.0.0rc1

which broke Prefect on Dask. The symptom is errors like this when a flow finishes:

distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/Users/jeff/.local/share/virtualenvs/jellyfish-u52nBq9x/lib/python3.7/site-packages/distributed/protocol/core.py", line 106, in loads
    header = msgpack.loads(header, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key

Reverting to

msgpack 0.6.x

solved the issue. Easy to repro on

prefect 0.8.1

and

dask 2.8.1

and

dask 2.9.1

. Not sure if this is something wonky in my environment or something real…

Hey Jeff - thanks for calling this out! I’ll dig into this a little further and see what I can learn

Interesting to consider. I saw this both in AWS (where the dask processes and the prefect process use the same image) as well as running on my local machine, where all processes share a virtualenv. That said, it’s entirely possible something was going on that I was unaware of…