Jeff Brainerd

01/09/2020, 2:22 AM
Hi all — I’m totally new to Prefect and Dask, but ran into an issue today that burned a few hours and wanted to make others aware. I inadvertantly upgraded the dependency
which broke Prefect on Dask. The symptom is errors like this when a flow finishes:
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/Users/jeff/.local/share/virtualenvs/jellyfish-u52nBq9x/lib/python3.7/site-packages/distributed/protocol/", line 106, in loads
    header = msgpack.loads(header, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
Reverting to
msgpack 0.6.x
solved the issue. Easy to repro on
prefect 0.8.1
dask 2.8.1
dask 2.9.1
. Not sure if this is something wonky in my environment or something real…
:upvote: 3

Chris White

01/09/2020, 3:16 AM
Hey Jeff - thanks for calling this out! I’ll dig into this a little further and see what I can learn
👍 2
Were you running a full cluster when you encountered this error? The reason I ask is that I believe this error could be a symptom of having differing versions of packages on your machine vs. the workers

Jeff Brainerd

01/09/2020, 6:17 PM
Interesting to consider. I saw this both in AWS (where the dask processes and the prefect process use the same image) as well as running on my local machine, where all processes share a virtualenv. That said, it’s entirely possible something was going on that I was unaware of…