Really sorry to keep bombarding the channel with so many questions - I think this will be the last one for the day. My question is: when you dockerize your flow, does this docker image replace the image of your dask workers for the purpose of executing your flow? I was under the impression that it does. However, I am getting this error w/ a custom library when running the flow from Prefect Server.

ModuleNotFoundError: No module named 'alternative_data_pipelines'

Note that the package

alternative_data_pipelines

exists in my docker file and has been pip-installed. No issues with the health checker - everything has been registered successfully and runs ok locally and from the docker container.

Hi @Riley Hun the docker image does not automatically replace the image used for your dask workers. If you use something like the DaskKubernetesEnvironment it does due to it spinning up a cluster for each run but otherwise you will have to ensure your dask workers have their needed dependencies installed

What about for Dask Gateway?

I’m not very familiar with Dask Gateway, maybe the devs over on their repo would be able to help there 🙂

I also wasn't able to overcome the

ModuleNotFoundError

. I've installed the custom package to the docker file of the dask workers successfully. I then SSH-ed into the dask workers on GKE and did a

pip list

to see if the package was there, and it was.

I am facing a similar problem. I ran dask-workers locally in a directory where I have file

somefile.py

but dask says no module named somefile.