Greg Roche
11/11/2020, 10:05 AMflow.run()
, but registering the flow on Prefect Server and triggering the flow from there fails because of module errors. I guess I'm just misunderstanding python's importing logic (I'm by no means an expert...) so a basic working example that I can build on would really help me get off the ground.my_project/
├── README.txt
├── __init__.py
├── requirements.txt
├── setup.py
├── etl
│ ├── __init__.py
│ ├── foo
│ │ ├── __init__.py
│ │ ├── foo_flow.py
│ │ ├── config
│ │ │ ├── __init__.py
│ │ │ └── foo_config.py
│ ├── shared
│ │ ├── __init__.py
│ │ └── sftp_utils.py
│ └── bar
│ ├── __init__.py
│ ├── bar_flow.py
│ ├── config
│ │ ├── bar_config.py
│ └── test
│ ├── __init__.py
│ └── test_bar_flow.py
├── .venv
├── start_agent.bat
I want to have two flows, foo_flow.py
and bar_flow.py
, each of which use some common logic imported from sftp_utils.py
, and have one agent launched from start_agent.bat
which can trigger both of the flows. My main pain points are:
• what to put into setup.py
so that running pip install -e .
in the root dir allows scripts to import from each other
• what importing syntax to use in (e.g.) foo_flow.py
to get logic from sftp_utils.py
Currently when I register the flow with Prefect Server, then start the local agent with --show-flow-logs
, the CLI spams error messages, starting with ModuleNotFoundError: No module named 'setuptools._distutils'
then repeating this over and over:
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 4>>
Traceback (most recent call last):
File "c:\my_project\.venv\lib\site-packages\tornado\ioloop.py", line 907, in _run
return self.callback()
File "c:\cmy_project\.venv\lib\site-packages\distributed\nanny.py", line 414, in memory_monitor
process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'
ale
11/11/2020, 11:05 AM$PYTHONPATH
so that they are visible to the flow when they run in Prefect ServerZanie
Greg Roche
11/11/2020, 3:36 PMZanie
Greg Roche
11/11/2020, 3:38 PMZanie
pip install -e .
with my module. Using docker storage would allow you to setup an environment alongside your flow so I recommend doing that in production but it depends on your setup!Greg Roche
11/12/2020, 8:21 AMfoo
flow in foo_flow.py
, then running python etl\foo\foo_flow.py
from the root folder to register the flow in Prefect Server, rather than having a register.py
file in the project root folder to handle the flow registration. In addition I didn't have my setup.py
file configured correctly by not specifying the etl
folder as a package. This meant I had to resort to PYTHONPATH hacks and other silliness to try to import from sibling folders, this way is so much cleaner and more elegant and extensible.
Michael, please let me know a way to buy you a couple of beers remotely as thanks :DZanie
Dave
11/21/2020, 11:13 PMSagun Garg
12/28/2020, 5:52 AMStep 6/10 : RUN mkdir -p /opt/prefect/
---> Using cache
---> 9b9f5132ea1a
Step 7/10 : COPY z-etl-flow.flow /opt/prefect/flows/z-etl-flow.prefect
---> fb8c3ce3bda9
Step 8/10 : COPY z-mr-flow.flow /opt/prefect/flows/z-mr-flow.prefect
---> 08102086f4ef
Step 9/10 : COPY healthcheck.py /opt/prefect/healthcheck.py
---> 725cca0c7872
Step 10/10 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/z-etl-flow.prefect", "/opt/prefect/flows/z-mr-flow.prefect"]' '(3, 7)'
---> Running in 0edd438d422e
Beginning health checks...
System Version check: OK
/opt/prefect/healthcheck.py:147: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
flows = cloudpickle_deserialization_check(flow_file_paths)
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 147, in <module>
flows = cloudpickle_deserialization_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
flows.append(cloudpickle.load(f))
ModuleNotFoundError: No module named 'prefectflows'
Removing intermediate container 0edd438d422e
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/z-etl-flow.prefect", "/opt/prefect/flows/z-mr-flow.prefect"]' '(3, 7)'' returned a non-zero code: 1
Traceback (most recent call last):
File "register_flows.py", line 36, in <module>
storage = storage.build()
File "/Users/sagungargs/Documents/portcast/flows/.venv/lib/python3.7/site-packages/prefect/storage/docker.py", line 360, in build
self._build_image(push=push)
File "/Users/sagungargs/Documents/portcast/flows/.venv/lib/python3.7/site-packages/prefect/storage/docker.py", line 428, in _build_image
"Your docker image failed to build! Your flow might have "
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
Zanie
COPY
for prefectflows
or RUN pip install ./prefectflows
in your logs