Hi, I am trying to make a flow run on my API but I...
# prefect-community
f
Hi, I am trying to make a flow run on my API but I get the following error from the UI :
Copy code
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
I don't understand the problem as I don't have any relative import in my code and the latter is pretty simple : https://github.com/flavienbwk/prefect-docker-compose/blob/main/scripts/weather.py
c
This is a new one for me; this is most likely related to your agent configuration
Thanks for linking to your repo! It’s helpful to have the context. Can you share where your Flow is stored on the machine? You should be able to find this information in the UI on the Flow page
f
Is this what you need ? (
/root/.prefect/flows
) ?
c
Yea; interesting that no
path
is displayed; so my rough takeaway is that your agent is looking for your Flow in a path that is
/root/.prefect/flows
but unable to access that, but I can’t 100% explain it. If you exec into your agent’s docker container, can you confirm that the flow exists at the right location?
f
So I've exec into my agent's docker container : the
/root/.prefect/flows
directory didn't even exist. I created it, re-run the flow and am still having the same error. Nothing is present in the
/root/.prefect/flows
directory.
c
your setup is not one I’ve personally seen before, so it’s hard to say. Let me explain what’s happening and maybe it will help you self-debug: - the agent queries the API and finds a flow run ID that needs to be run - the agent then submits the CLI command
prefect execute flow-run
to run in a subprocess - this code is then run, which is where your error arises from: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/execute.py#L74-L93 Here’s how you might drill down into what step is failing: - change the labels on your Flow in the UI (so your agent won’t pick anything up), and create a new flow run - within your agent’s docker container, run each of the python commands in that file I linked to, starting with this line: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/execute.py#L56 My guess is that you’ll see this error arise either at this line:
Copy code
flow = storage.get_flow(storage.flows[flow_data.name])
note that that line attempts to retrieve your flow code from the location that it is stored in, which in your case is
/root/.prefect/flows
another thing you could do that is more direct is to run:
Copy code
prefect.core.flow.Flow.load(flow_location)
with your Flow’s location
sorry for the stream of consciousness but I see what’s happening
the main thing is that your flow’s file (where it’s stored) needs to be accessible by your Agent - it is currently not, but the error is obscuring that. I’ll update the error message to be more clear
f
Wow OK I see the idea. So the code is not "saved to the API and then redistributed to the agents". The API just shares a kind of ID that is expected to match a directory present in the agent's
/root/.prefect/flows
. Is it correct ?
c
Yea exactly! It is a very unique design and can be initially confusing, but it’s designed that way for a very explicit reason (check out this post for more info: https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296)
f
OK I see !
E*xecution in your cloud; orchestration in ours*
Yeah it's not very easy to understand and maybe the documentation should include a step by step guide to understand how Prefect's storage system work. But I'm convinced it's a bright idea, so I will deep dive into the documentation and try to understand more about Prefect storage system.
👍 2
I've come to the most basic solution for my issue. My "scripts" creating the flow will share the same Docker volume than my agents. Currently it works. I didn't understand yet the storage system with Docker images, and am afraid of the performances but I will investigate that solution. Maybe that for high performance needs, HDFS might be a solution.
👍 1