Hi, when I run Prefect server with my custom docker image, I get the following error

Failed to load and execute flow run: ModuleNotFoundError("No module named 'pandas'")

I don’t think it’s coming from missing pandas module. When I log into my docker image, I have no problem importing pandas. I’ve installed all the required packages in the base environment. I’ve also tried loading all the packages in the conda environment and that the conda environment is loaded upon docker image is loaded. Is this some python path issue? What I don’t understand is that i’ve done this several times before and i didn’t have this issue until now

check this Discourse topic https://discourse.prefect.io/t/when-i-run-my-flow-i-see-an-error-failed-to-load-and-exe[…]derror-no-module-named-users-username-what-is-happening/33

I don’t think it’s easy to activate a conda environment in an image. How are you doing it? Simply doing

conda activate

doesn’t work

FROM prefecthq/prefect:latest

WORKDIR $HOME/myproject
ADD . .

SHELL [ "/bin/bash", "--login", "-c" ]

RUN apt-get clean &&  apt-get update && apt-get install -y curl vim

ENV MINICONDA_VERSION latest
ENV CONDA_DIR $HOME/miniconda3

RUN curl -LO "<https://repo.anaconda.com/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh>" \
    && bash Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -p $CONDA_DIR -b
RUN rm Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh

ENV PATH=$CONDA_DIR/bin:$PATH
RUN echo ". $CONDA_DIR/etc/profile.d/conda.sh" >> ~/.profile
RUN conda init bash

RUN conda env update -n base -f environment.yml --prune && \
    conda clean --all -f -y

This is the copy of my Dockerfile

I don’t think this will work in setting the environment during runtime? Are you in the right conda environment when you exec? Have you seen this article?

unfortunately i do because i need a package that only exists in conda

yeah i was wondering about if this will apply to the base environment as well, but I personally can’t guess if it will work. what is the command you use to start the image?

bash

?

yea i’m using bash

I can test in a couple of hours on conda and bash and see if base is respected

thank you

Testing now

great!

So I definitely replicated. Works when I exec but not with a Prefect Flow. I am somewhat positive the article i showed above is what we are running into here and i’ve given the workaround for people but never tried it myself

i haven’t tried using entrypoint. I’ll give it a try. I also reproduced the issue in the docker image. I guess it also confirms it’s related.. it’s using a different python version when i execute flow

(dockerimage)# prefect execute flow-run
No module named 'pandas'
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 92, in flow_run
    raise exc
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run
    flow = storage.get_flow(flow_data.name)
  File "/usr/local/lib/python3.7/site-packages/prefect/storage/gitlab.py", line 106, in get_flow
    file_contents=contents.decode(), flow_name=flow_name
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 89, in extract_flow_from_file
    exec(contents, exec_vals)
  File "<string>", line 7, in <module>
ModuleNotFoundError: No module named 'pandas'

the python version where i have everything installed should be python 3.9 and not 3.7

Yeah the Dockerfile you have closely resembles the article’s “What does not work #1”. I honestly don’t get the “why?” so much though

so entrypoint.sh didn’t help…

I don’t know if you can even change. Did a quick search for now but couldn’t find anything. A bit confused, the Python path seems to be for your flow directory right? Or is it also related to the package management?

yea but when i do

which python

, i get miniconda python path. In miniconda python path, i have all the packages i need installed as expected

do you use sudo in your docker setup?