jpuris
03/27/2023, 3:21 PM/tmp
where it would basically double the snapshot size compared to venv
it is copied from.
..more info in the thread$ du -hs /opt/my_prefect_project/venv
509M /opt/my_prefect_project/venv
But when a i.e. hello world
flow is run on it, it will copy the whole venv, which makes sense! It simply does not know what libs may or may not be required... I'm ok with that.
However.. when it does get snapshotted to /tmp
for the flow run, it doubles in size!
$ du -h -d5 /tmp/*prefect/ | sort -rh | head -1
1.1G /tmp/tmpv17bxswaprefect/venv
So far I've narrowed it down to the fact it does not understand symlinks in the original venv..
$ ls -la venv/
total 24
drwxrwxr-x 5 ubuntu ubuntu 4096 Feb 9 09:30 .
drwxrwxr-x 8 ubuntu ubuntu 4096 Mar 14 14:26 ..
drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 14 09:22 bin
drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 9 09:30 include
drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 9 09:30 lib
lrwxrwxrwx 1 ubuntu ubuntu 3 Feb 9 09:30 lib64 -> lib
-rw-rw-r-- 1 ubuntu ubuntu 70 Feb 9 09:30 pyvenv.cfg
While on snapshot side..
$ du -h -d5 /tmp/*prefect/ | sort -rh | grep lib
509M /tmp/tmpv17bxswaprefect/venv/lib64
509M /tmp/tmpv17bxswaprefect/venv/lib
which results in total of 2x venv dir sizeno disk space remaining
errors 😞Zanie
03/27/2023, 3:35 PMjpuris
03/27/2023, 3:36 PMI think we explicitly follow symlinksHi @Zanie, does that mean this is not intended behaviour? I can produce a MRE, if necessary. To my knowledge, the lib and lib64 symlink exists only for Debian bases distributions 🤷
Zanie
03/27/2023, 3:37 PMIf symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree and the metadata of the original links will be copied as far as the platform allows; if false or omitted, the contents and metadata of the linked files are copied to the new tree.
When symlinks is false, if the file pointed by the symlink doesn’t exist, an exception will be added in the list of errors raised in anexception at the end of the copy process. You can set the optional _ignore_dangling_symlinks_ flag to true if you want to silence this exception. Notice that this option has no effect on platforms that don’t supportError
.os.symlink()
False
jpuris
03/27/2023, 3:40 PMZanie
03/27/2023, 3:41 PM