hey guys, i'm trying to get prefect running on kub...
# ask-community
l
hey guys, i'm trying to get prefect running on kubernetes. i've packaged my entire python package into docker container, and i'm able to run the flows in the container just fine locally using
docker run
. however, when i deploy the flows to kube and use prefect cloud to trigger a run, i get the following error:
Copy code
Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'my_package\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
i'm using GCS as the storage. i'm also wondering if/why i need to use GCS as a storage -- doesn't the execution environment have access to the flows? why do they even need to be pickled? thanks in advance! 🙂
a
This page provides more details: https://discourse.prefect.io/t/when-i-run-my-flow-i-see-an-error-failed-to-load-and-exe[…]derror-no-module-named-users-username-what-is-happening/33 In short: your Docker container usually has code dependencies (e.g. your custom modules or things like requests or pandas), while storage references your flow file
if you want a single docker image used as both code and flow storage, check out those examples https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_no_build
l
thanks anna! are tasks serialized too or are they considered dependencies that should be deployed into the execution environment?
in this particular case, why is the module not found? i have deployed my_package into the execution environment. so the flow should be able to import that module when it is unpickled?
k
Tasks are not serialized, just the metadata around them, but the code is loaded in from the Storage. What RunConfig are you using? DockerRun?
l
oh, i didn't specify any RunConfig
so probably UniversalRun which is the default from my understanding
k
So what is your execution environment? Local agent?
l
kubernetes
k
So it’s using the default Prefect image to run the Flow?
l
i have a Dockerfile that uses prefecthq/prefect:latest as the base image
basically just pip installing my package
k
Where did you pass the Dockerfile? I thought you where on GCS Storage?
l
the dockerfile builds all my code into a python package. that is deployed to kubernetes
the flows use GCS as their storage
k
You need to supply that image to
KubernetesRun
otherwise it won’t know what image to use and will pull the default Prefect image and run your Flow on top of that. The Flow does not run in the same pod as the agent. The agent spins up a new Kubernetes job so the execution environment is different
l
Ah i see, thanks for the clarification. I will make that change and let you know the outcome. Thanks!
it works, thanks for your help!
👍 1