flavienbwk

    flavienbwk

    1 year ago
    Hi, I am trying to make a flow run on my API but I get the following error from the UI :
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
    I don't understand the problem as I don't have any relative import in my code and the latter is pretty simple : https://github.com/flavienbwk/prefect-docker-compose/blob/main/scripts/weather.py
    Chris White

    Chris White

    1 year ago
    This is a new one for me; this is most likely related to your agent configuration
    Thanks for linking to your repo! It’s helpful to have the context. Can you share where your Flow is stored on the machine? You should be able to find this information in the UI on the Flow page
    flavienbwk

    flavienbwk

    1 year ago
    Is this what you need ? (
    /root/.prefect/flows
    ) ?
    Chris White

    Chris White

    1 year ago
    Yea; interesting that no
    path
    is displayed; so my rough takeaway is that your agent is looking for your Flow in a path that is
    /root/.prefect/flows
    but unable to access that, but I can’t 100% explain it. If you exec into your agent’s docker container, can you confirm that the flow exists at the right location?
    flavienbwk

    flavienbwk

    1 year ago
    So I've exec into my agent's docker container : the
    /root/.prefect/flows
    directory didn't even exist. I created it, re-run the flow and am still having the same error. Nothing is present in the
    /root/.prefect/flows
    directory.
    Chris White

    Chris White

    1 year ago
    your setup is not one I’ve personally seen before, so it’s hard to say. Let me explain what’s happening and maybe it will help you self-debug: - the agent queries the API and finds a flow run ID that needs to be run - the agent then submits the CLI command
    prefect execute flow-run
    to run in a subprocess- this code is then run, which is where your error arises from: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/execute.py#L74-L93 Here’s how you might drill down into what step is failing:- change the labels on your Flow in the UI (so your agent won’t pick anything up), and create a new flow run - within your agent’s docker container, run each of the python commands in that file I linked to, starting with this line: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/execute.py#L56 My guess is that you’ll see this error arise either at this line:
    flow = storage.get_flow(storage.flows[flow_data.name])
    note that that line attempts to retrieve your flow code from the location that it is stored in, which in your case is
    /root/.prefect/flows
    another thing you could do that is more direct is to run:
    prefect.core.flow.Flow.load(flow_location)
    with your Flow’s location
    sorry for the stream of consciousness but I see what’s happening
    the main thing is that your flow’s file (where it’s stored) needs to be accessible by your Agent - it is currently not, but the error is obscuring that. I’ll update the error message to be more clear
    flavienbwk

    flavienbwk

    1 year ago
    Wow OK I see the idea. So the code is not "saved to the API and then redistributed to the agents". The API just shares a kind of ID that is expected to match a directory present in the agent's
    /root/.prefect/flows
    . Is it correct ?
    Chris White

    Chris White

    1 year ago
    Yea exactly! It is a very unique design and can be initially confusing, but it’s designed that way for a very explicit reason (check out this post for more info: https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296)
    flavienbwk

    flavienbwk

    1 year ago
    OK I see !
    Execution in your cloud; orchestration in ours
    Yeah it's not very easy to understand and maybe the documentation should include a step by step guide to understand how Prefect's storage system work. But I'm convinced it's a bright idea, so I will deep dive into the documentation and try to understand more about Prefect storage system.
    I've come to the most basic solution for my issue. My "scripts" creating the flow will share the same Docker volume than my agents. Currently it works. I didn't understand yet the storage system with Docker images, and am afraid of the performances but I will investigate that solution. Maybe that for high performance needs, HDFS might be a solution.