r

    Romain

    1 year ago
    Hi folks, Have an issue with running a flow using a Kubernetes Agent, and a Module storage. So here is what I do. The agent docker image is based on prefecthq/prefect:0.14.19, and in my dockerfile, I have the following folder hierarchy:
    /A/
      B/
         flows/
             my_flow.py
    in my_flow.py, imagine something like that:
    def get_flow():
        with Flow('my_flow', storage=Module("B.flows.my_flow:get_flow")) as flow:
            ....
        return flow
    In the dockerfile, I ensure that the
    PYTHONPATH
    env var holds the folder
    A
    so that I can import
    <http://B.flows.my|B.flows.my>_flow
    setting this at the end of the DockerFile:
    ENV PYTHONPATH "${PYTHONPATH}:/A"
    After that I register this flow with a
    KubernetesRun
    and a DaskExecutor:
    flow.run_config = KubernetesRun()
    flow.executor = DaskExecutor()
    flow.register(project_name='my_project',
                  idempotency_key=flow.serialized_hash())
    Then from a prefect server, I trigger the flow run, but I get the following error:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'A'")
    I don't really get why my module is not found. I have tested a local deployment using docker-compose with a local agent (running in the compose stack), and it was working fine. So I am missing something here. Any ideas?
    Shall the module be also available on the prefect server apollo pod?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Romain! Yes you need it installed and available in the execution location.
    r

    Romain

    1 year ago
    But I thought that in prefect , the server was somewhat agnostic of the execution code. And only the agent was required to know the code. How would it work if I would be using prefect cloud and a Module storage then?
    Kevin Kho

    Kevin Kho

    1 year ago
    With Docker storage, the container is downloaded and executed on the agent. For Module storage, the module would need to exist on the execution side also because it doesn’t download it from a location. This is true for Cloud as well.