r

    Ruben Sik

    9 months ago
    Hi all, does anyone have experience in imports when running a prefect docker agent? While on my local docker image my imports work fine, when running a flow via a docker agent, modules cannot be found?
    More info: dockerfile:
    FROM python:3.8-slim-buster
    WORKDIR /app
    COPY requirements.txt ./requirements.txt
    RUN pip install "prefect[azure, gitlab]"
    RUN pip install -r requirements.txt
    COPY . .
    ENV PYTHONPATH "${PYTHONPATH}:/app"
    Furthermore we are using gitlab storage and our folder structure looks like:
    |__app     
    |  └── __init__.py
    |  └── flows
    |  └── └── __init__.py
    |  └── └── test_flow.py
    |  └── └── helper_script.py
    |  └── └── utils
    |  └── └── └── __init__.py
    |  └── └── └── helper_script.py
    |  └── utils
    |  └── └── __init__.py
    |  └── └── helper_script.py
    We've copied the helper_script.py to test multiple location to try various imports. When we run os.listdir() in our flow script "test_flow.py", the logger seems to show our utils folder is in the current working directory. This os.listdir() results in [utils, flows] which should mean an import possibily of "import utils.helper_script" (whichs works on command line of the docker image but not when running the flow via a docker agent). Are we overlooking something?
    Anna Geller

    Anna Geller

    9 months ago
    @Ruben Sik in general, it would be easier for you if you would add
    setup.py
    and make
    utils
    a package. This way your package will be importable anywhere (in a virtual environment, Docker image etc) and you don’t have to manually add directories to the PYTHONPATH and it makes your code much cleaner. This post explains how to do it. I also have an example Dockerfile and setup.py in this repo.
    r

    Ruben Sik

    9 months ago
    Tnx @Anna Geller I read about this strategy this morning and gave it a go. Following the guide, the pip environment shows the created 'utils' as in the foto below. When inspecting this package, it seems as there is content in the package. For the steps i've followed Kevin's guide with this structure: app/ ├── utils/ │ ├── init.py │ ├── uDatabase.py │ ├── uDatalake.py │ ├── uEmail.py │ ├── uSharepoint.py ├── flows/ │ ├── flow.py ├── requirements.txt ├── Dockerfile └── setup.py However when running the docker agent and executing my flow.py which includes the imports of utils/uEmail, no import seems to work. Neither: import utils from utils import uEmail import utils.uEmail app.utils.uEmail any idea?
    Anna Geller

    Anna Geller

    9 months ago
    @Ruben Sik I think I know what is the issue. You have the
    app
    directory also on your local, right?
    in that case the copy command for you would be:
    COPY app/. .
    otherwise you’re docker image gets this utils directory:
    /app/app/utils
    so the import would have to be:
    from app.app.utils import ...
    If you want to start from scratch, you can try the approach from this repo. It’s based on the prefect base image and also contains a flow_utilities folder. You could follow the same structure and even pretty much reuse the Dockerfile:
    FROM prefecthq/prefect:0.15.10-python3.8
    RUN /usr/local/bin/python -m pip install --upgrade pip
    WORKDIR /opt/prefect
    COPY utils/ /opt/prefect/utils/
    COPY requirements.txt .
    COPY setup.py .
    RUN pip install .
    COPY flows/ /opt/prefect/flows/
    note that copying the flows into a Docker images is only relevant when using Docker storage with
    stored_as_script=True
    . If you wanna use this pattern, check out this example
    r

    Ruben Sik

    9 months ago
    Tnx Anna, when using your dockerfile as a template it seems to work now. Error seems to indeed be due to the working directory/ copying steps before!