Anders Lindstrom

    Anders Lindstrom

    10 months ago
    I have a basic question about the use of
    Module
    storage in a Docker container. I've been trying to get it working for half a day now and I'm stumped. I'm sure it's a very simple problem but I'm unable to find any working examples as a basis. So I have a flow that is installed (via
    pip install
    ) in a Docker image. The flow is called
    flows.docker_2
    . It looks like this:
    import prefect
    from prefect import task, Flow
    from prefect.run_configs import DockerRun
    from prefect.storage import Module
    
    @task
    def say_hello():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("Hello, docker 2!")
    
    with Flow("docker_2_flow") as flow:
        flow.storage = Module("flows.docker_2")
        flow.run_config = DockerRun(
            image="prefect_docker_2_image",
            host_config={
                "auto_remove": False
            }
        )
        say_hello()
    
    if __name__ == "__main__":
        flow.register(project_name="docker_2_project", labels=["docker_flows"])
    I am able to register the flow by executing it inside the container:
    docker run --network prefect-server --rm --name prefect_docker_2 -it prefect_docker_2:latest python -m flows.docker_2
    So I know that the module is installed and can be run inside the container. However, when I run the flow, the module is not found. The docker agent reports:
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
    ModuleNotFoundError: No module named 'flows'
    Is there something else I have to do (other than pip installing the module) to make the flow code visible?
    Kevin Kho

    Kevin Kho

    10 months ago
    You were on StackOverflow with the
    --expose
    question right? Nice to see you in the Slack. The attempt here looks good. My guess would be that your image might have multiple Python installations and the wrong one is being used? Maybe you can try
    exec
    -ing into the container and doing
    where python
    Anders Lindstrom

    Anders Lindstrom

    10 months ago
    Yes. That's me.
    Thanks for the encouragement. I'm almost there πŸ™‚
    😳 Embarrassing. I was looking in the wrong place. I thought my understanding of Prefect was wrong but it was simply a mismatch between the image I was building and the image referred to in the flow. It works now. Thanks for your help.
    Kevin Kho

    Kevin Kho

    10 months ago
    Oh well done! Glad it’s working πŸ‘