Hi folks, I'm having issues with ModuleNotFoundErr...
# prefect-community
Hi folks, I'm having issues with ModuleNotFoundError when running a flow from a local agent. I have a project structure like this:
Copy code
     |  |--__init__.py
     |  |--foo_flow.py
     |  |--__init__.py
     |  |--bar_flow.py
I import modules from
by appending the etl folder to sys.path before importing
. When I run any of these flows with
they work fine. However, when I do
instead and then start a local agent, from any of the directories listed above, all flow runs initiated from the server to the agent fail instantly with
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'shared_utils'"
. According to a stackoverflow answer this is because the agent's python path doesn't include
but it still fails when I run the agent with
--import-path "C:\project\etl"
. I've tried registering the flow from every directory listed above, and tried starting the agent from every directory listed above, and also tried passing every directory listed above as an
argument, and I get exactly the same error every time. Can anybody please point out what I'm missing?
Hi Greg,
is not the correct name of the CLI flag, it’s actually
, or the more abbreviated
Let me know if that solves it for you!
Sorry, that was a typo here, fixed now - I was using
in my script. After a bunch more testing, I've found a combination of options that doesn't cause the flow to error out immediately: both registering the flow and starting the agent from
now just makes the flow hang forever with status "running flow" in the server UI. If I run the agent with
I see a torrent of errors mentioning
ModuleNotFoundError: No module named 'setuptools._distutils'
before the following error repeats endlessly over and over until I kill the agent:
Copy code
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 4>>
Traceback (most recent call last):
  File "c:\project\.venv\lib\site-packages\tornado\ioloop.py", line 907, in _run
    return self.callback()
  File "c:\project\.venv\lib\site-packages\distributed\nanny.py", line 414, in memory_monitor
    process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'
🤯 this seems like a deep rooted path / environment issue. There’s a chance that this PR would resolve it for you: https://github.com/PrefectHQ/prefect/pull/3361 if you’re interested in testing that out
Thanks for the tip, I will try that out :)
@Greg Roche Did you end up finding a solution? I am having the same problem. Thanks!
ok if I start the agent with
prefect agent local start --show-flow-logs
from the package directory works but if i add any
option that point to the directory of the package (even if i am inside the package) I get the
missing module
@simone the root fix for this for me was using absolute imports for all local code, and installing my code locally as an editable module with
pip install -e .
. There's another thread here https://prefect-community.slack.com/archives/CL09KU1K7/p1605089116281400 where a Prefect dev provided me with an example project structure to get me started, that was super helpful and will probably point you in the right direction too.
@Greg Roche Thanks a lot for the tip! I will look at the project structure and see how I can setup my code. Thanks!