Greg Roche
09/22/2020, 2:35 PM|--project
|-.venv
|-etl
|--__init__.py
|--foo
| |--__init__.py
| |--foo_flow.py
|--bar
| |--__init__.py
| |--bar_flow.py
|--shared_utils
|--__init__.py
|--utils.py
In foo_flow.py
and bar_flow.py
I import modules from shared_utils
by appending the etl folder to sys.path before importing utils.py
. When I run any of these flows with flow.run()
they work fine.
However, when I do flow.register()
instead and then start a local agent, from any of the directories listed above, all flow runs initiated from the server to the agent fail instantly with Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'shared_utils'"
. According to a stackoverflow answer this is because the agent's python path doesn't include project/etl
but it still fails when I run the agent with --import-path "C:\project\etl"
. I've tried registering the flow from every directory listed above, and tried starting the agent from every directory listed above, and also tried passing every directory listed above as an --import-project
argument, and I get exactly the same error every time. Can anybody please point out what I'm missing?Chris White
--import-project
is not the correct name of the CLI flag, it’s actually --import-path
, or the more abbreviated -p
Let me know if that solves it for you!Greg Roche
09/22/2020, 3:58 PM--import-path
in my script.
After a bunch more testing, I've found a combination of options that doesn't cause the flow to error out immediately: both registering the flow and starting the agent from project/etl
now just makes the flow hang forever with status "running flow" in the server UI. If I run the agent with -f
I see a torrent of errors mentioning ModuleNotFoundError: No module named 'setuptools._distutils'
before the following error repeats endlessly over and over until I kill the agent:
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 4>>
Traceback (most recent call last):
File "c:\project\.venv\lib\site-packages\tornado\ioloop.py", line 907, in _run
return self.callback()
File "c:\project\.venv\lib\site-packages\distributed\nanny.py", line 414, in memory_monitor
process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'
Chris White
Greg Roche
09/22/2020, 4:04 PMsimone
12/19/2020, 11:17 AMprefect agent local start --show-flow-logs
from the package directory works but if i add any -p
option that point to the directory of the package (even if i am inside the package) I get the missing module
errorGreg Roche
12/24/2020, 8:27 AMpip install -e .
. There's another thread here https://prefect-community.slack.com/archives/CL09KU1K7/p1605089116281400 where a Prefect dev provided me with an example project structure to get me started, that was super helpful and will probably point you in the right direction too.simone
01/04/2021, 7:07 AM