Thread
#prefect-community
    t

    Trever Mock

    2 years ago
    Hey all, happy Tuesday!   I'm trying to import additional generated Python files and use them as part of my Prefect Workflow. Everything works fine locally, but when I push up the flow to a docker repo and try to run it using a Kubernetes agent, I'm getting the following error from cloudpickle:   ModuleNotFoundError: No module named 'test_pb2'   The Kubernetes agent spins up a job in our cluster, and this error comes from that job. It works fine if the flow doesn't reference any other custom modules.   Here's what I've tried so far, but it all results in the same error (note that the healthcheck.py passes just fine in all these cases when the flow is pushed to docker storage):   1.       Setting PYTHONPATH to point to test_pb2 and other files. If I pull the container, PYTHONPATH is correct; however, when the kubernetes agent spins up a job using that container, it doesn't appear like it's set anymore. Is it getting overridden somewhere?   2.       Turning the generated files into a package installable with pip and then installing it as part of the Dockerfile that creates the docker storage. If I pull the container, the package appears to be installed successfully. Still the same error when run by the kubernetes agent though.   Any ideas on other things I could try? Is there a preferred way to include other Python files as part of a prefect flow?   Any ideas/help is greatly appreciated!!
    j

    josh

    2 years ago
    Hey @Trever Mock could you provide a reproducible example of your flow / example of how it is accessing
    test_pb2
    ? Preferably as an issue on the repo as it makes code debugging easier 🙂
    t

    Trever Mock

    2 years ago
    Thanks for the response 😀! I'll try to put together a Github issue that better describes the steps I'm using. I'll reply here when I do. It might take a bit, I've got to strip out some proprietary stuff from the workflow I'm using right now.
    @josh it seems it was an issue on our end with how our kube cluster was set up. It's working now 🙂. Thanks again for such a quick reply!