jars
06/17/2020, 3:23 AMfrom lib.package1 import fn1
from lib.package2 import fn2
These correspond to files alongside my flow like: lib/package1.py
and lib/package2.py
.
When calling flow.register()
it looks like my Docker Storage healthcheck fails because it cannot find the lib module...
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 135, in <module>
flows = cloudpickle_deserialization_check(flow_file_path)
File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
flows.append(cloudpickle.load(f))
ModuleNotFoundError: No module named 'lib'
Any tips on how to make that work?Chris White
06/17/2020, 3:26 AM$PATH
Arsenii
06/17/2020, 3:31 AMfiles
argument.
Pass something like like files_to_be_copied = {i: f"/root/lib/extra_python_files/{i.name}" for i in extra_python_files}
where extra_python_files
includes the local paths to the files you need copied over.
Then, use env_vars
argument in Docker to pass something like {"PYTHONPATH": "/root/lib/extra_python_files"}
Inside the flow, you'll maybe have to adjust the from ... import ...
line. During debugging, you may use relative imports, but when the flow runs they'll fail.$PATH
... The only solution, it seems, is to hardcode paths to these files inside the flow definition and replicate the folder structure exactly in Docker(..., files=...)
.
If anyone has better ideas, I'd love to hear themChris White
06/17/2020, 4:15 AMDockerfile
with:
COPY ./my-package-contents /place-in-docker
RUN pip install /place-in-docker
jars
06/17/2020, 4:27 AMNo module named 'flow'
It doesn't seem like I can fix it the same way.
I noticed something peculiar though, while looking at these Docs while investigating: https://docs.prefect.io/core/advanced_tutorials/local-debugging.html#locally-check-your-flow-s-docker-storage
When running print(built_storage.flows)
, it returns as {}
My code is, at it's barest, something like this. Is this the right idea?
# my real flow lives in here...
from flow import flow
from prefect.environments.storage.docker import Docker
flow.storage = Docker(
registry_url=registry_url,
dockerfile='Dockerfile',
image_name='my_image',
image_tag='0.0.0',
env_vars={}
)
built_storage = flow.storage.build(push=False)
# returns {} ?
print(built_storage.flows)
Chris White
06/17/2020, 4:29 AMflow
that isn’t present within the imageJacob Blanco
06/17/2020, 4:32 AMChris White
06/17/2020, 4:34 AMjars
06/17/2020, 4:41 AMfrom flow import flow
worked... well, it obviously worked because the flow.py was inside my local directory. I understand... the health check needed access to it as well. I added it's location to PYTHONPATH and all is well.Chris White
06/17/2020, 4:41 AMMarvin
06/17/2020, 4:44 AMJacob Blanco
06/17/2020, 5:16 AMmiko
06/17/2020, 9:31 AMChris White
06/17/2020, 3:40 PMmiko
06/17/2020, 3:42 PM