https://prefect.io logo
Channels
Powered by
m

Mia

04/05/2022, 2:12 PM
Hi, when I run Prefect server with my custom docker image, I get the following error
Copy code
Failed to load and execute flow run: ModuleNotFoundError("No module named 'pandas'")
I don’t think it’s coming from missing pandas module. When I log into my docker image, I have no problem importing pandas. I’ve installed all the required packages in the base environment. I’ve also tried loading all the packages in the conda environment and that the conda environment is loaded upon docker image is loaded. Is this some python path issue? What I don’t understand is that i’ve done this several times before and i didn’t have this issue until now
a

Anna Geller

04/05/2022, 2:22 PM
if you need help troubleshooting, it would help if you share your Dockerfie and your storage + run config
k

Kevin Kho

04/05/2022, 2:23 PM
I don’t think it’s easy to activate a conda environment in an image. How are you doing it? Simply doing 
conda activate
doesn’t work
m

Mia

04/05/2022, 2:28 PM
Copy code
FROM prefecthq/prefect:latest

WORKDIR $HOME/myproject
ADD . .

SHELL [ "/bin/bash", "--login", "-c" ]

RUN apt-get clean &&  apt-get update && apt-get install -y curl vim

ENV MINICONDA_VERSION latest
ENV CONDA_DIR $HOME/miniconda3

RUN curl -LO "<https://repo.anaconda.com/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh>" \
    && bash Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -p $CONDA_DIR -b
RUN rm Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh

ENV PATH=$CONDA_DIR/bin:$PATH
RUN echo ". $CONDA_DIR/etc/profile.d/conda.sh" >> ~/.profile
RUN conda init bash

RUN conda env update -n base -f environment.yml --prune && \
    conda clean --all -f -y
This is the copy of my Dockerfile
k

Kevin Kho

04/05/2022, 2:53 PM
I don’t think this will work in setting the environment during runtime? Are you in the right conda environment when you exec? Have you seen this article?
But also though, do you really need conda in the image? Since the image already provides environment isolation
💯 1
m

Mia

04/05/2022, 3:38 PM
unfortunately i do because i need a package that only exists in conda
but pandas should be installed also in base environment. So I don’t think the issue is with the conda environment not being activated
k

Kevin Kho

04/05/2022, 4:10 PM
yeah i was wondering about if this will apply to the base environment as well, but I personally can’t guess if it will work. what is the command you use to start the image? 
bash
?
m

Mia

04/05/2022, 4:23 PM
yea i’m using bash
k

Kevin Kho

04/05/2022, 4:23 PM
I can test in a couple of hours on conda and bash and see if base is respected
m

Mia

04/05/2022, 4:26 PM
thank you
k

Kevin Kho

04/05/2022, 7:21 PM
Testing now
m

Mia

04/05/2022, 7:42 PM
great!
I’ve also tried fixing the python path but that didn’t work ENV PYTHONPATH=“${PYTHONPATH}:/myproject/flows”
k

Kevin Kho

04/05/2022, 8:10 PM
So I definitely replicated. Works when I exec but not with a Prefect Flow. I am somewhat positive the article i showed above is what we are running into here and i’ve given the workaround for people but never tried it myself
Could you maybe install a wheel instead by copying it in if it lets you go to pip? Cuz conda is pretty hard. Anyway, I believe people have tried using the ENTRYPOINT method mentioned in that article. Have you tried it? Copy pasting it here to be specific
Copy code
#!/bin/bash --login
# The --login ensures the bash configuration is loaded,
# enabling Conda.

# Enable strict mode.
set -euo pipefail
# ... Run whatever commands ...

# Temporarily disable strict mode and activate conda:
set +euo pipefail
conda activate myenv

# Re-enable strict mode:
set -euo pipefail
m

Mia

04/05/2022, 8:24 PM
i haven’t tried using entrypoint. I’ll give it a try. I also reproduced the issue in the docker image. I guess it also confirms it’s related.. it’s using a different python version when i execute flow
Copy code
(dockerimage)# prefect execute flow-run
No module named 'pandas'
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 92, in flow_run
    raise exc
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run
    flow = storage.get_flow(flow_data.name)
  File "/usr/local/lib/python3.7/site-packages/prefect/storage/gitlab.py", line 106, in get_flow
    file_contents=contents.decode(), flow_name=flow_name
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 89, in extract_flow_from_file
    exec(contents, exec_vals)
  File "<string>", line 7, in <module>
ModuleNotFoundError: No module named 'pandas'
the python version where i have everything installed should be python 3.9 and not 3.7
k

Kevin Kho

04/05/2022, 8:27 PM
Yeah the Dockerfile you have closely resembles the article’s “What does not work #1”. I honestly don’t get the “why?” so much though
m

Mia

04/05/2022, 8:27 PM
so entrypoint.sh didn’t help…
i don’t get it why 
prefect execute flow-run
uses a specific python version and not the one in the path
how do i change the path that prefect execute uses? when i explicitly download pandas package to python3.7 (the one prefect execute uses), the error seems to go away
k

Kevin Kho

04/05/2022, 8:42 PM
I don’t know if you can even change. Did a quick search for now but couldn’t find anything. A bit confused, the Python path seems to be for your flow directory right? Or is it also related to the package management?
Can try to see what I find a bit later tonight
m

Mia

04/05/2022, 8:50 PM
yea but when i do 
which python
, i get miniconda python path. In miniconda python path, i have all the packages i need installed as expected
but it seems prefect is using a different python that’s not in the path
a

Anna Geller

04/05/2022, 9:27 PM
do you use sudo in your docker setup?
32 Views
Powered by