https://prefect.io logo
j

José Duarte

09/14/2022, 1:24 PM
Hey all, I am trying to run flows using Docker as infrastructure. I’ve set the pull policy to never because I’m currently performing tests and the image is not public — in other words, I am forcing the agent to use the local ones. So far, so good. The issue arises when running the flow, the container will try to run
python -m prefect.engine flow_uuid
, it fetches the flow but then tries to copy files into the same folder, in this case
deployment.path
is the
cwd
&
pwd
which in turn means that
.
resolves to
deployment.path
— making the
filesystems.LocalFileSystem.get_directory
copy the files into themselves (through
shutil.copytree
) which ends up in a HUGE error like so:
Copy code
[<redacted for brevity>, '.', 'c', 'f', 'g', "'", '>', ' ', 'a', 'n', 'd', ' ', "'", '.', '/', '.', 'i', 's', 'o', 'r', 't', '.', 'c', 'f', 'g', "'", ' ', 'a', 'r', 'e', ' ', 't', 'h', 'e', ' ', 's', 'a', 'm', 'e', ' ', 'f', 'i', 'l', 'e', '<', 'D', 'i', 'r', 'E', 'n', 't', 'r', 'y', ' ', "'", '.', 'f', 'l', 'a', 'k', 'e', '8', "'", '>', ' ', 'a', 'n', 'd', ' ', "'", '.', '/', '.', 'f', 'l', 'a', 'k', 'e', '8', "'", ' ', 'a', 'r', 'e', ' ', 't', 'h', 'e', ' ', 's', 'a', 'm', 'e', ' ', 'f', 'i', 'l', 'e']
With regards to Prefect’s code, I have a solution, just check if both paths match and don’t copy in that case, but I’d like to get some feedback from the Prefect team before reporting a bug and sending in a fix — i.e. if I am doing something wrong.
1
I also don’t get why Prefect is copying the files at all, it seems to be an unnecessary extra step
Nevermind, theres another issue on GH with the same problem I’ve created a PR https://github.com/PrefectHQ/prefect/pull/6824