Thread
#prefect-server
    m

    Maria

    5 months ago
    Hi, when I run Prefect server with my custom docker image, I get the following error
    Failed to load and execute flow run: ModuleNotFoundError("No module named 'pandas'")
    I don’t think it’s coming from missing pandas module. When I log into my docker image, I have no problem importing pandas. I’ve installed all the required packages in the base environment. I’ve also tried loading all the packages in the conda environment and that the conda environment is loaded upon docker image is loaded. Is this some python path issue? What I don’t understand is that i’ve done this several times before and i didn’t have this issue until now
    if you need help troubleshooting, it would help if you share your Dockerfie and your storage + run config
    Kevin Kho

    Kevin Kho

    5 months ago
    I don’t think it’s easy to activate a conda environment in an image. How are you doing it? Simply doing
    conda activate
    doesn’t work
    m

    Maria

    5 months ago
    FROM prefecthq/prefect:latest
    
    WORKDIR $HOME/myproject
    ADD . .
    
    SHELL [ "/bin/bash", "--login", "-c" ]
    
    RUN apt-get clean &&  apt-get update && apt-get install -y curl vim
    
    ENV MINICONDA_VERSION latest
    ENV CONDA_DIR $HOME/miniconda3
    
    RUN curl -LO "<https://repo.anaconda.com/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh>" \
        && bash Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -p $CONDA_DIR -b
    RUN rm Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh
    
    ENV PATH=$CONDA_DIR/bin:$PATH
    RUN echo ". $CONDA_DIR/etc/profile.d/conda.sh" >> ~/.profile
    RUN conda init bash
    
    RUN conda env update -n base -f environment.yml --prune && \
        conda clean --all -f -y
    This is the copy of my Dockerfile
    Kevin Kho

    Kevin Kho

    5 months ago
    I don’t think this will work in setting the environment during runtime? Are you in the right conda environment when you exec? Have you seen this article?
    But also though, do you really need conda in the image? Since the image already provides environment isolation
    m

    Maria

    5 months ago
    unfortunately i do because i need a package that only exists in conda
    but pandas should be installed also in base environment. So I don’t think the issue is with the conda environment not being activated
    Kevin Kho

    Kevin Kho

    5 months ago
    yeah i was wondering about if this will apply to the base environment as well, but I personally can’t guess if it will work. what is the command you use to start the image?
    bash
    ?
    m

    Maria

    5 months ago
    yea i’m using bash
    Kevin Kho

    Kevin Kho

    5 months ago
    I can test in a couple of hours on conda and bash and see if base is respected
    m

    Maria

    5 months ago
    thank you
    Kevin Kho

    Kevin Kho

    5 months ago
    Testing now
    m

    Maria

    5 months ago
    great!
    I’ve also tried fixing the python path but that didn’t workENV PYTHONPATH=“${PYTHONPATH}😕myproject/flows”
    Kevin Kho

    Kevin Kho

    5 months ago
    So I definitely replicated. Works when I exec but not with a Prefect Flow. I am somewhat positive the article i showed above is what we are running into here and i’ve given the workaround for people but never tried it myself
    Could you maybe install a wheel instead by copying it in if it lets you go to pip? Cuz conda is pretty hard. Anyway, I believe people have tried using the ENTRYPOINT method mentioned in that article. Have you tried it? Copy pasting it here to be specific
    #!/bin/bash --login
    # The --login ensures the bash configuration is loaded,
    # enabling Conda.
    
    # Enable strict mode.
    set -euo pipefail
    # ... Run whatever commands ...
    
    # Temporarily disable strict mode and activate conda:
    set +euo pipefail
    conda activate myenv
    
    # Re-enable strict mode:
    set -euo pipefail
    m

    Maria

    5 months ago
    i haven’t tried using entrypoint. I’ll give it a try. I also reproduced the issue in the docker image. I guess it also confirms it’s related.. it’s using a different python version when i execute flow
    (dockerimage)# prefect execute flow-run
    No module named 'pandas'
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 92, in flow_run
        raise exc
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run
        flow = storage.get_flow(flow_data.name)
      File "/usr/local/lib/python3.7/site-packages/prefect/storage/gitlab.py", line 106, in get_flow
        file_contents=contents.decode(), flow_name=flow_name
      File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 89, in extract_flow_from_file
        exec(contents, exec_vals)
      File "<string>", line 7, in <module>
    ModuleNotFoundError: No module named 'pandas'
    the python version where i have everything installed should be python 3.9 and not 3.7
    Kevin Kho

    Kevin Kho

    5 months ago
    Yeah the Dockerfile you have closely resembles the article’s “What does not work #1”. I honestly don’t get the “why?” so much though
    m

    Maria

    5 months ago
    so entrypoint.sh didn’t help…
    i don’t get it why
    prefect execute flow-run
    uses a specific python version and not the one in the path
    how do i change the path that prefect execute uses? when i explicitly download pandas package to python3.7 (the one prefect execute uses), the error seems to go away
    Kevin Kho

    Kevin Kho

    5 months ago
    I don’t know if you can even change. Did a quick search for now but couldn’t find anything. A bit confused, the Python path seems to be for your flow directory right? Or is it also related to the package management?
    Can try to see what I find a bit later tonight
    m

    Maria

    5 months ago
    yea but when i do
    which python
    , i get miniconda python path. In miniconda python path, i have all the packages i need installed as expected
    but it seems prefect is using a different python that’s not in the path
    Anna Geller

    Anna Geller

    5 months ago
    do you use sudo in your docker setup?