Hi all, does anyone have experience in imports whe...
# ask-communityr
Ruben Sik
12/10/2021, 8:23 AMHi all, does anyone have experience in imports when running a prefect docker agent? While on my local docker image my imports work fine, when running a flow via a docker agent, modules cannot be found?
Ruben Sik
12/10/2021, 8:32 AMMore info:
dockerfile:
Copy code
FROM python:3.8-slim-buster
WORKDIR /app
COPY requirements.txt ./requirements.txt
RUN pip install "prefect[azure, gitlab]"
RUN pip install -r requirements.txt
COPY . .
ENV PYTHONPATH "${PYTHONPATH}:/app" Copy code
|__app
| └── __init__.py
| └── flows
| └── └── __init__.py
| └── └── test_flow.py
| └── └── helper_script.py
| └── └── utils
| └── └── └── __init__.py
| └── └── └── helper_script.py
| └── utils
| └── └── __init__.py
| └── └── helper_script.pya
Anna Geller
12/10/2021, 10:07 AM
@Ruben Sik in general, it would be easier for you if you would add
setup.pyutilsr
Ruben Sik
12/10/2021, 1:37 PMTnx @Anna Geller I read about this strategy this morning and gave it a go. Following the guide, the pip environment shows the created 'utils' as in the foto below. When inspecting this package, it seems as there is content in the package. For the steps i've followed Kevin's guide with this structure:
app/
├── utils/
│ ├── init.py
│ ├── uDatabase.py
│ ├── uDatalake.py
│ ├── uEmail.py
│ ├── uSharepoint.py
├── flows/
│ ├── flow.py
├── requirements.txt
├── Dockerfile
└── setup.py
However when running the docker agent and executing my flow.py which includes the imports of utils/uEmail, no import seems to work. Neither:
import utils
from utils import uEmail
import utils.uEmail
app.utils.uEmail
any idea?
a
Anna Geller
12/10/2021, 1:52 PM
@Ruben Sik I think I know what is the issue.
You have the
appAnna Geller
12/10/2021, 1:53 PM
in that case the copy command for you would be:
Copy code
COPY app/. .Anna Geller
12/10/2021, 1:55 PM
otherwise you’re docker image gets this utils directory:
Copy code
/app/app/utils Copy code
from app.app.utils import ...Anna Geller
12/10/2021, 1:57 PM
If you want to start from scratch, you can try the approach from this repo. It’s based on the prefect base image and also contains a flow_utilities folder. You could follow the same structure and even pretty much reuse the Dockerfile:
Copy code
FROM prefecthq/prefect:0.15.10-python3.8
RUN /usr/local/bin/python -m pip install --upgrade pip
WORKDIR /opt/prefect
COPY utils/ /opt/prefect/utils/
COPY requirements.txt .
COPY setup.py .
RUN pip install .
COPY flows/ /opt/prefect/flows/Anna Geller
12/10/2021, 1:59 PM
note that copying the flows into a Docker images is only relevant when using Docker storage with
stored_as_script=Truer
Ruben Sik
12/10/2021, 3:02 PMTnx Anna, when using your dockerfile as a template it seems to work now. Error seems to indeed be due to the working directory/ copying steps before!
🙌 1
5 Views