https://prefect.io logo
Title
d

Daniel Davee

06/28/2021, 6:11 PM
Can I use the docker storage to load an entire folder?
Docker(
    files={
        # absolute path source -> destination in image
        "/Users/me/code/my_mods/": "/my_mods",
        
    },
    env_vars={
        # append modules directory to PYTHONPATH
        "PYTHONPATH": "$PYTHONPATH:modules/"
    },
)
k

Kevin Kho

06/28/2021, 6:49 PM
Hi @Daniel Davee, it seems like you can provide a directory and it will use
shutil.copytree
to copy the folder. Code . Are you seeing different behavior?
d

Daniel Davee

06/28/2021, 7:20 PM
Yes
k

Kevin Kho

06/28/2021, 7:23 PM
Do you get an error or nothing happens?
What version are you on also?
d

Daniel Davee

06/28/2021, 7:41 PM
0.14.19
And I'm just looking to automate the oneboard process as much as possible
k

Kevin Kho

06/28/2021, 7:45 PM
This should work for 0.14.19. Sorry I’m a bit confused. You tried passing the folder and it didn’t work? Or are you just asking for now?
d

Daniel Davee

06/28/2021, 7:46 PM
I have a dockerfile template, and I'm getting this error. /opt/prefect/healthcheck.py:152: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules https://docs.prefect.io/api/latest/storage.html#docker  flows = cloudpickle_deserialization_check(flow_file_paths) Traceback (most recent call last):  File "/opt/prefect/healthcheck.py", line 152, in <module>   flows = cloudpickle_deserialization_check(flow_file_paths)  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check   flows.append(cloudpickle.loads(flow_bytes)) ModuleNotFoundError: No module named 'recommender'
k

Kevin Kho

06/28/2021, 7:50 PM
Are you using the
Dockerfile
or the
Docker(files=…)
? I think the path of the place running might be different from what you are expecting. This will be easier if you make a minimal package by adding a
setup.py
to your project and then
pip install
the dependency in your Dockerfile. Would you like an example of that?
d

Daniel Davee

06/28/2021, 7:58 PM
yes please
k

Kevin Kho

06/28/2021, 7:59 PM
Check the
Dockerfile
and
setup.py
here
d

Daniel Davee

06/28/2021, 7:59 PM
thank you
👍 1
k

Kevin Kho

06/28/2021, 7:59 PM
You want the
pip -install -e .
and the
setup.py
to install that dependency as a package