https://prefect.io logo
d

David Hlavaty

09/01/2022, 12:33 PM
Hi, great stuff in adding support for flows inside Docker image. It's much appreciate feature. Thanks Now a bit of feedback - my docker images are based on
python:3-9slim
and I then install all my dependencies including Prefect from a lock file. I prefer this over using the official Prefect images as it means there is a single source of truth for which version of Prefect I am using. I had my image "misconfigured" by setting working directory to
/opt/prefect/flows
. This then resulted in a cryptic message from
shutil
when running my flows as it tried to copy files from the flow source directory (
/opt/prefect/flows
by default) to a working directory (
/opt/prefect/flows
) which are the same. Would be good if Prefect detected this and did not attempt to copy the files. Or at least checked if the working directory is the same and failed with helpful error message.
a

Adam Brusselback

09/01/2022, 12:54 PM
Literally just opened an issue for this: https://github.com/PrefectHQ/prefect/issues/6666
@David Hlavaty just wanted to see if we have anything in common in our setups, are you using Python 3.9 or newer?
d

David Hlavaty

09/01/2022, 4:03 PM
See my message above - this is inside python-3.9-slim docker image, and specifically related to Prefect trying to copy the source code when
src dir == dest dir
In your case, the problem originates from using Python 3.7 as behaviour of https://docs.python.org/3.7/library/shutil.html#shutil.copytree
Recursively copy an entire directory tree rooted at src, returning the destination directory. The destination directory, named by dst, must not already exist; it will be created as well as missing parent directories. Permissions and times of directories are copied with
copystat()
, individual files are copied using
shutil.copy2()
.
If you follow stack-trace from your issue, it will lead you here
Copy code
if sys.version_info < (3, 8):
            shutil.copytree(from_path, local_path)
        else:
            shutil.copytree(from_path, local_path, dirs_exist_ok=True)
Notice that for Python 3.7 and lower it does not tolerate existing directory (it can't as per docs above). Given that
local_path
default to current working directory, I don't think this can ever work since the current working directory exists by definition.
a

Adam Brusselback

09/01/2022, 4:52 PM
Yup, got it. Thanks for the info as to exactly why it fails, as I didn't have my pwd set to the directory I was trying get my flows from in my systemd service file...