https://prefect.io logo
Title
j

Jakub Vedral

01/29/2021, 12:54 PM
Another thing is that I am getting a strange error for this simple Flow:
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'C'")
Can anybody tell me some hint where to look for a cause? googling did not help... 😞 More details in thread.
g

Greg Roche

01/29/2021, 4:02 PM
Would you mind please moving all the lengthy stacktrace etc. into this thread, rather than being in the main post? This makes it more comfortable for those browsing the slack channel.
j

Jakub Vedral

01/29/2021, 4:03 PM
yep, sorry for that 😐 UI error:
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'C'")
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/prefect/__main__.py", line 4, in <module>
    cli()
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/prefect/cli/execute.py", line 90, in flow_run
    raise exc
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/prefect/cli/execute.py", line 67, in flow_run
    flow = storage.get_flow(flow_data.name)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/prefect/storage/local.py", line 104, in get_flow
    return extract_flow_from_module(module_str=flow_location)
  File "/home/jakub_vedral/prefect-env/lib/python3.7/site-packages/prefect/utilities/storage.py", line 131, in extract_flow_from_module
    module = importlib.import_module(module_name)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'C'
flow definition:
from datetime import timedelta
from prefect import Flow
from prefect.schedules import IntervalSchedule
from prefect import task

@task(name="test")
def test():
    return "Hi!"



with Flow("Test Run",IntervalSchedule(interval=timedelta(hours=4)),validate =True) as f:

    test()



f.register(project_name="test",labels=["production"])
Prefect server is running on my GCP instance under virtualenv  -  Linux prefect-prod 4.19.0-13-cloud-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 pip packages of server instancne:
cached-property==1.5.2
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
click==7.1.2
cloudpickle==1.6.0
croniter==0.3.37
dask==2021.1.1
distributed==2021.1.1
docker==4.4.1
HeapDict==1.0.1
idna==2.10
importlib-resources==5.1.0
marshmallow==3.10.0
marshmallow-oneofschema==2.1.0
msgpack==1.0.2
mypy-extensions==0.4.3
natsort==7.1.1
pendulum==2.1.2
pkg-resources==0.0.0
prefect==0.14.5
psutil==5.8.0
pycparser==2.20
pygit2==1.5.0
python-box==5.2.0
python-dateutil==2.8.1
python-slugify==4.0.1
pytz==2020.5
pytzdata==2020.1
PyYAML==5.4.1
requests==2.25.1
six==1.15.0
sortedcontainers==2.3.0
tabulate==0.8.7
tblib==1.7.0
text-unidecode==1.3
toml==0.10.2
toolz==0.11.1
tornado==6.1
urllib3==1.26.3
websocket-client==0.57.0
zict==2.0.0
zipp==3.4.0
z

Zanie

01/29/2021, 4:42 PM
Hi @Jakub Vedral — I don’t see how this error could be thrown from the code you provided. It’s entering a code path that I’m pretty sure is only encountered when
Local
storage is explicitly defined.
Are you sure this is the same flow code that’s erroring?
j

Jakub Vedral

01/29/2021, 4:47 PM
Yes, I am sure. I think it is related to the server environment and running prefect server from venv, but i dont know how to debug that. When i run it locally from my windows machine, it runs fine.
z

Zanie

01/29/2021, 4:48 PM
Can you try adding
flow.storage = Local()
and reregistering?
It’s trying to import it from a module which should not be the default behavior — I’m not even sure where it’s getting a module path from
Can you also provide the output of
prefect diagnostics
on the machine where you are registering the flow?
j

Jakub Vedral

01/29/2021, 4:57 PM
prefect diagnostics: { "config_overrides": { "context": { "secrets": false } }, "env_vars": [], "system_information": { "platform": "Linux-4.19.0-13-cloud-amd64-x86_64-with-debian-10.7", "prefect_backend": "server", "prefect_version": "0.14.5", "python_version": "3.7.3" } }
setting flow.storage = Local() did not help, still the same error
z

Zanie

01/29/2021, 5:04 PM
Can you print
flow.storage.path
?
Nevermind that’s not what we want
So this module-not-found error indicates that the flow path is
C:\….
which is a windows path and it’s not found on the file system where your agent is running so it tries to load it from a module
Have you registered this flow on a windows machine?
j

Jakub Vedral

01/29/2021, 5:07 PM
yep i have a debian linux machine as a server on GCP and registered the flow from my windows machine
z

Zanie

01/29/2021, 5:08 PM
Ah okay, and where are you running your agent?
j

Jakub Vedral

01/29/2021, 5:08 PM
my agent is also on the linux server
it has the same pip packages as my local windows machine
z

Zanie

01/29/2021, 5:10 PM
Alrighty — so the issue here is that your flow is using local storage because you aren’t setting a storage so it’s being written to your windows machine’s file system then your agent can’t find it because it’s running on your server.
Generally this is avoided by setting labels on flows that are local to the hostname of the system that has the flow file on it so the agent won’t try to pick it up but it looks like you’re setting labels explicitly.
j

Jakub Vedral

01/29/2021, 5:11 PM
yep, i realized that if i dont make the labels of agent and my flow the same it will not start. I can see now why is that implicitly taken care of 🙂
z

Zanie

01/29/2021, 5:12 PM
You’ll need to switch to some form of remote storage so the agent can access the flow and you’ll be good!
j

Jakub Vedral

01/29/2021, 8:06 PM
Well I changed the storage to the remote and everything works. Thanks so much for your help 🙂