I am running into an issue importing helper files,...
# prefect-server
I am running into an issue importing helper files, deployed on Kubernetes, using KubernetesAgent and DaskExecutor…file structure is:
Copy code
I am getting
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'my_tasks'")
when trying to run the flow in
. I have a feeling that what’s happening is the
file only lives on the Agent or Job Pod, and not the Dask pods…any way around this?
Hey @Sam Werbalowsky, this is because Flow storage only contains the file with the Flow. These aren;t serialized along with it because of how
works. You need to package these into a Docker image and then use that Docker image for your Dask pods so that you have these tasks available on the Dask cluster.
Had a feeling that was the case. Thanks!
If it helps you, I have a tutorial on the packing into the container part, but I figure you might be familiar if you are on k8s
Please send it over! This is my first real dig into k8s actually so any resources help
This is for the packaging in Docker
@Kevin Kho I know it’s late, so not expecting a response til the morning - is there a way to do this without having to use a
pip install .
with a requirements and setup file? I copied the files over succesfully to my image, but I still am getting the
. I think it has to do with the way the actual execution occurs, but I can’t quite figure it out.
I can exec into the pod and it the import works fine, so I’m not sure why it wouldn’t work on the agent.
Oh yeah I forgot to test. Sorry, will give it a shot in a bit
I did get it working with the
FYI …it’s not so bad I am just looking for something simpler if possible 😄
no rush
Yeah I think it’s just about the PYTHONPATH and I haven’t tried it myself so would be good to also
So I believe I got this working. Here is my Dockerfile
Copy code
FROM prefecthq/prefect:latest



ADD . .
My directory structure is the same as my blog with:
Copy code
├── components/
│   ├── __init__.py
│   ├── componentA.py
│   ├── componentB.py
├── workflow/
│   ├── flow.py
├── requirements.txt
├── Dockerfile
└── setup.py
and then I can build locally with
docker build . -t test-path:latest
and then run with
docker run -it test-path:latest sh
Then you can open a Python interpreter
Copy code
> import components
> from components import componentA
and this will work (but of course that’s because you start in the app directory but if you
cd ..
cd usr
- or any other folder and repeat the interpreter and the import, it will work because the
folder is in the PYTHONPATH so it can import the contents