I am running into an issue importing helper files deployed o

Question

I am running into an issue importing helper files deployed on Kubernetes using KubernetesAgent and DaskExecutor file structure is ```flows | my flow py | my tasks py | helpers py``` I am getting `Failed to load and execute Flow s environment ModuleNotFoundError No module named my tasks ` when trying to run the flow in `my flow` I have a feeling that what s happening is the `my tasks` file only lives on the Agent or Job Pod and not the Dask pods any way around this

Kevin Kho · Answer

Hey < Sam Werbalowsky> this is because Flow storage only contains the file with the Flow These aren t serialized along with it because of how `cloudpickle` works You need to package these into a Docker image and then use that Docker image for your Dask pods so that you have these tasks available on the Dask cluster

Sam Werbalowsky · Answer

Had a feeling that was the case Thanks

Kevin Kho · Answer

If it helps you I have a tutorial on the packing into the container part but I figure you might be familiar if you are on k8s

Sam Werbalowsky · Answer

Please send it over This is my first real dig into k8s actually so any resources help

Kevin Kho · Answer

<https medium com the prefect blog the simple guide to productionizing data workflows with docker 31a5aae67c0a|This> is for the packaging in Docker

Sam Werbalowsky · Answer

< Kevin Kho> I know it s late so not expecting a response til the morning is there a way to do this without having to use a `pip install ` with a requirements and setup file I copied the files over succesfully to my image but I still am getting the `ModuleNotFoundError` I think it has to do with the way the actual execution occurs but I can t quite figure it out

Sam Werbalowsky · Answer

I can exec into the pod and it the import works fine so I m not sure why it wouldn t work on the agent

Kevin Kho · Answer

Oh yeah I forgot to test Sorry will give it a shot in a bit

Sam Werbalowsky · Answer

I did get it working with the `setup py` FYI it s not so bad I am just looking for something simpler if possible smile

Sam Werbalowsky · Answer

no rush

Kevin Kho · Answer

Yeah I think it s just about the PYTHONPATH and I haven t tried it myself so would be good to also

Kevin Kho · Answer

So I believe I got this working Here is my Dockerfile ```FROM prefecthq prefect latest ENV PYTHONPATH $ PYTHONPATH app WORKDIR app ADD ``` My directory structure is the same as my blog with ```mypackage components init py componentA py componentB py workflow flow py requirements txt Dockerfile setup py``` and then I can build locally with `docker build t test path latest` and then run with `docker run it test path latest sh`

Kevin Kho · Answer

Then you can open a Python interpreter ```python gt import components gt from components import componentA``` and this will work but of course that s because you start in the app directory but if you `cd ` `cd usr` or any other folder and repeat the interpreter and the import it will work because the `app` folder is in the PYTHONPATH so it can import the contents