I’m experiencing an issue with unpickling when a flow is registered from inside a conda environment. Registering the flow directly from a

virtualenv

which contains the necessary dependencies works and the flow runs as expected. But registering the flow from a conda environment I receive this unpickling error when running.

Hey @Sean Harkins! This seems tricky. Are you using the conda environment to register but then running with Docker?

I hope this isn’t anything exceedingly tricky. We have been registering flows using virtualenvs and the flows are executed using an agent and dask cluster running in ECS (the worker image uses a conda environment which is activated as part of the base image) and this works fine. But for our CI we need to have a wide variety of potential dependencies which our flows might require so I tried to use the same worker image with the conda environment active for registering flows and now I am seeing this error.

Are the versions pinned? Did you output the versions of all packages and compare?

Identical

Do you need it pickled? You can try storing as a script too

We have been consistently using the pickled approach successfully so I would like to at least understand why it is failing in this case before making a switch.

That makes sense. This error indicates some version mismatch right? You could also try running with debug logs?

So interestingly this module in

xarray

was just recently refactored https://github.com/pydata/xarray/commit/6e14df62f0b01d8ca5b04bd0ed2b5ee45444265d#diff-b73383e3adc1135bda2af03eaa6[…]5a64e23c193e4f351f6bcdc17fc0d30f but we have everything pinned to

0.18.0

prior to this change so I’m not sure how this updated version might be creeping in.

Just wondering how you're sure the conda environment is being activated? Some people have issues with activating conda in Docker. Would your setup just fail entirely if the environment we're activated?

@Kevin Kho The issue was a package that is not published in conda and therefore needed to be

pip

installed into the conda environment. This package was upgrading the

xarray

version on the image registering our flow. Thanks for rubber ducking me through it 😄

Oh jeez. How long did this take to find? A full day?

No. Thankfully I was able to track it down pretty quickly after your suggestion 🙇

Oh that’s good to hear. Nice job!