Thread
#prefect-community
    r

    Riley Hun

    2 years ago
    Really sorry to keep bombarding the channel with so many questions - I think this will be the last one for the day. My question is: when you dockerize your flow, does this docker image replace the image of your dask workers for the purpose of executing your flow? I was under the impression that it does. However, I am getting this error w/ a custom library when running the flow from Prefect Server.
    ModuleNotFoundError: No module named 'alternative_data_pipelines'
    Note that the package
    alternative_data_pipelines
    exists in my docker file and has been pip-installed. No issues with the health checker - everything has been registered successfully and runs ok locally and from the docker container.
    j

    josh

    2 years ago
    Hi @Riley Hun the docker image does not automatically replace the image used for your dask workers. If you use something like the DaskKubernetesEnvironment it does due to it spinning up a cluster for each run but otherwise you will have to ensure your dask workers have their needed dependencies installed
    r

    Riley Hun

    2 years ago
    What about for Dask Gateway?
    j

    josh

    2 years ago
    I’m not very familiar with Dask Gateway, maybe the devs over on their repo would be able to help there 🙂
    r

    Riley Hun

    2 years ago
    I also wasn't able to overcome the
    ModuleNotFoundError
    . I've installed the custom package to the docker file of the dask workers successfully. I then SSH-ed into the dask workers on GKE and did a
    pip list
    to see if the package was there, and it was.
    Rahat Zaman

    Rahat Zaman

    1 year ago
    I am facing a similar problem. I ran dask-workers locally in a directory where I have file
    somefile.py
    but dask says no module named somefile.