Hi folks, I'm having issues with ModuleNotFoundError when running a flow from a local agent. I have a project structure like this:

|--project
   |-.venv
   |-etl
     |--__init__.py
     |--foo
     |  |--__init__.py
     |  |--foo_flow.py
     |--bar
     |  |--__init__.py
     |  |--bar_flow.py
     |--shared_utils
        |--__init__.py
        |--utils.py

In

foo_flow.py

and

bar_flow.py

I import modules from

shared_utils

by appending the etl folder to sys.path before importing

utils.py

. When I run any of these flows with

flow.run()

they work fine. However, when I do

flow.register()

instead and then start a local agent, from any of the directories listed above, all flow runs initiated from the server to the agent fail instantly with

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'shared_utils'"

. According to a stackoverflow answer this is because the agent's python path doesn't include

project/etl

but it still fails when I run the agent with

--import-path "C:\project\etl"

. I've tried registering the flow from every directory listed above, and tried starting the agent from every directory listed above, and also tried passing every directory listed above as an

--import-project

argument, and I get exactly the same error every time. Can anybody please point out what I'm missing?

Hi Greg,

--import-project

is not the correct name of the CLI flag, it’s actually

--import-path

, or the more abbreviated

-p

Let me know if that solves it for you!

Sorry, that was a typo here, fixed now - I was using

--import-path

in my script. After a bunch more testing, I've found a combination of options that doesn't cause the flow to error out immediately: both registering the flow and starting the agent from

project/etl

now just makes the flow hang forever with status "running flow" in the server UI. If I run the agent with

-f

I see a torrent of errors mentioning

ModuleNotFoundError: No module named 'setuptools._distutils'

before the following error repeats endlessly over and over until I kill the agent:

tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 4>>
Traceback (most recent call last):
  File "c:\project\.venv\lib\site-packages\tornado\ioloop.py", line 907, in _run
    return self.callback()
  File "c:\project\.venv\lib\site-packages\distributed\nanny.py", line 414, in memory_monitor
    process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'

🤯 this seems like a deep rooted path / environment issue. There’s a chance that this PR would resolve it for you: https://github.com/PrefectHQ/prefect/pull/3361 if you’re interested in testing that out

Thanks for the tip, I will try that out :)

@Greg Roche Did you end up finding a solution? I am having the same problem. Thanks!

@simone the root fix for this for me was using absolute imports for all local code, and installing my code locally as an editable module with

pip install -e .

. There's another thread here https://prefect-community.slack.com/archives/CL09KU1K7/p1605089116281400 where a Prefect dev provided me with an example project structure to get me started, that was super helpful and will probably point you in the right direction too.

@Greg Roche Thanks a lot for the tip! I will look at the project structure and see how I can setup my code. Thanks!