Thread
#prefect-community
    Jeff Brainerd

    Jeff Brainerd

    2 years ago
    Hi all — I’m totally new to Prefect and Dask, but ran into an issue today that burned a few hours and wanted to make others aware. I inadvertantly upgraded the dependency
    msgpack
    to
    v1.0.0rc1
    which broke Prefect on Dask. The symptom is errors like this when a flow finishes:
    distributed.protocol.core - CRITICAL - Failed to deserialize
    Traceback (most recent call last):
      File "/Users/jeff/.local/share/virtualenvs/jellyfish-u52nBq9x/lib/python3.7/site-packages/distributed/protocol/core.py", line 106, in loads
        header = msgpack.loads(header, use_list=False, **msgpack_opts)
      File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
    ValueError: tuple is not allowed for map key
    Reverting to
    msgpack 0.6.x
    solved the issue. Easy to repro on
    prefect 0.8.1
    and
    dask 2.8.1
    and
    dask 2.9.1
    . Not sure if this is something wonky in my environment or something real…
    Chris White

    Chris White

    2 years ago
    Hey Jeff - thanks for calling this out! I’ll dig into this a little further and see what I can learn
    Were you running a full cluster when you encountered this error? The reason I ask is that I believe this error could be a symptom of having differing versions of packages on your machine vs. the workers
    Jeff Brainerd

    Jeff Brainerd

    2 years ago
    Interesting to consider. I saw this both in AWS (where the dask processes and the prefect process use the same image) as well as running on my local machine, where all processes share a virtualenv. That said, it’s entirely possible something was going on that I was unaware of…