Sam Werbalowsky
09/23/2021, 11:09 PMflows
|____my_flow.py
|____my_tasks.py
|___helpers.py
I am getting Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'my_tasks'")
when trying to run the flow in my_flow
. I have a feeling that what’s happening is the my_tasks
file only lives on the Agent or Job Pod, and not the Dask pods…any way around this?Kevin Kho
09/24/2021, 2:12 AMcloudpickle
works. You need to package these into a Docker image and then use that Docker image for your Dask pods so that you have these tasks available on the Dask cluster.Sam Werbalowsky
09/24/2021, 3:55 AMKevin Kho
09/24/2021, 4:01 AMSam Werbalowsky
09/24/2021, 11:38 AMKevin Kho
09/24/2021, 1:45 PMSam Werbalowsky
09/27/2021, 11:51 PMpip install .
with a requirements and setup file? I copied the files over succesfully to my image, but I still am getting the ModuleNotFoundError
. I think it has to do with the way the actual execution occurs, but I can’t quite figure it out.Kevin Kho
09/28/2021, 1:48 AMSam Werbalowsky
09/28/2021, 1:48 AMsetup.py
FYI …it’s not so bad I am just looking for something simpler if possible 😄Kevin Kho
09/28/2021, 1:49 AMFROM prefecthq/prefect:latest
ENV PYTHONPATH "${PYTHONPATH}:/app"
WORKDIR /app
ADD . .
My directory structure is the same as my blog with:
mypackage/
├── components/
│ ├── __init__.py
│ ├── componentA.py
│ ├── componentB.py
├── workflow/
│ ├── flow.py
├── requirements.txt
├── Dockerfile
└── setup.py
and then I can build locally with docker build . -t test-path:latest
and then run with docker run -it test-path:latest sh
python
> import components
> from components import componentA
and this will work (but of course that’s because you start in the app directory but if you
cd ..
cd usr
- or any other folder
and repeat the interpreter and the import, it will work because the app
folder is in the PYTHONPATH so it can import the contents