https://prefect.io logo
Title
r

Riley Hun

08/12/2020, 8:12 PM
Really sorry to keep bombarding the channel with so many questions - I think this will be the last one for the day. My question is: when you dockerize your flow, does this docker image replace the image of your dask workers for the purpose of executing your flow? I was under the impression that it does. However, I am getting this error w/ a custom library when running the flow from Prefect Server.
ModuleNotFoundError: No module named 'alternative_data_pipelines'
Note that the package
alternative_data_pipelines
exists in my docker file and has been pip-installed. No issues with the health checker - everything has been registered successfully and runs ok locally and from the docker container.
j

josh

08/12/2020, 8:18 PM
Hi @Riley Hun the docker image does not automatically replace the image used for your dask workers. If you use something like the DaskKubernetesEnvironment it does due to it spinning up a cluster for each run but otherwise you will have to ensure your dask workers have their needed dependencies installed
r

Riley Hun

08/13/2020, 12:35 AM
What about for Dask Gateway?
j

josh

08/13/2020, 1:01 AM
I’m not very familiar with Dask Gateway, maybe the devs over on their repo would be able to help there 🙂
r

Riley Hun

08/13/2020, 2:16 AM
I also wasn't able to overcome the
ModuleNotFoundError
. I've installed the custom package to the docker file of the dask workers successfully. I then SSH-ed into the dask workers on GKE and did a
pip list
to see if the package was there, and it was.
r

Rahat Zaman

09/28/2020, 4:41 AM
I am facing a similar problem. I ran dask-workers locally in a directory where I have file
somefile.py
but dask says no module named somefile.