https://prefect.io
Join Slack
I am running into an issue importing helper files, deployed on Kubernetes, using KubernetesAgent and...
# prefect-server
s
I am running into an issue importing helper files, deployed on Kubernetes, using KubernetesAgent and DaskExecutor…file structure is:
Copy code
flows
|____my_flow.py
|____my_tasks.py
|___helpers.py
I am getting 
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'my_tasks'")
when trying to run the flow in 
my_flow
. I have a feeling that what’s happening is the 
my_tasks
file only lives on the Agent or Job Pod, and not the Dask pods…any way around this?
k
Hey @Sam Werbalowsky, this is because Flow storage only contains the file with the Flow. These aren;t serialized along with it because of how 
cloudpickle
works. You need to package these into a Docker image and then use that Docker image for your Dask pods so that you have these tasks available on the Dask cluster.
s
Had a feeling that was the case. Thanks!
k
If it helps you, I have a tutorial on the packing into the container part, but I figure you might be familiar if you are on k8s
s
Please send it over! This is my first real dig into k8s actually so any resources help
k
This is for the packaging in Docker
s
@Kevin Kho I know it’s late, so not expecting a response til the morning - is there a way to do this without having to use a 
pip install .
with a requirements and setup file? I copied the files over succesfully to my image, but I still am getting the 
ModuleNotFoundError
. I think it has to do with the way the actual execution occurs, but I can’t quite figure it out.
I can exec into the pod and it the import works fine, so I’m not sure why it wouldn’t work on the agent.
k
Oh yeah I forgot to test. Sorry, will give it a shot in a bit
s
I did get it working with the 
setup.py
FYI …it’s not so bad I am just looking for something simpler if possible 😄
no rush
k
Yeah I think it’s just about the PYTHONPATH and I haven’t tried it myself so would be good to also
So I believe I got this working. Here is my Dockerfile
Copy code
FROM prefecthq/prefect:latest

ENV PYTHONPATH "${PYTHONPATH}:/app"

WORKDIR /app

ADD . .
My directory structure is the same as my blog with:
Copy code
mypackage/
├── components/
│   ├── __init__.py
│   ├── componentA.py
│   ├── componentB.py
├── workflow/
│   ├── flow.py
├── requirements.txt
├── Dockerfile
└── setup.py
and then I can build locally with 
docker build . -t test-path:latest
and then run with 
docker run -it test-path:latest sh
Then you can open a Python interpreter
Copy code
python
> import components
> from components import componentA
and this will work (but of course that’s because you start in the app directory but if you 
cd ..
cd usr
- or any other folder and repeat the interpreter and the import, it will work because the 
app
folder is in the PYTHONPATH so it can import the contents
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by