jpuris03/27/2023, 3:21 PM
where it would basically double the snapshot size compared to
it is copied from. ..more info in the thread
But when a i.e.
$ du -hs /opt/my_prefect_project/venv 509M /opt/my_prefect_project/venv
flow is run on it, it will copy the whole venv, which makes sense! It simply does not know what libs may or may not be required... I'm ok with that. However.. when it does get snapshotted to
for the flow run, it doubles in size!
So far I've narrowed it down to the fact it does not understand symlinks in the original venv..
$ du -h -d5 /tmp/*prefect/ | sort -rh | head -1 1.1G /tmp/tmpv17bxswaprefect/venv
While on snapshot side..
$ ls -la venv/ total 24 drwxrwxr-x 5 ubuntu ubuntu 4096 Feb 9 09:30 . drwxrwxr-x 8 ubuntu ubuntu 4096 Mar 14 14:26 .. drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 14 09:22 bin drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 9 09:30 include drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 9 09:30 lib lrwxrwxrwx 1 ubuntu ubuntu 3 Feb 9 09:30 lib64 -> lib -rw-rw-r-- 1 ubuntu ubuntu 70 Feb 9 09:30 pyvenv.cfg
which results in total of 2x venv dir size
$ du -h -d5 /tmp/*prefect/ | sort -rh | grep lib 509M /tmp/tmpv17bxswaprefect/venv/lib64 509M /tmp/tmpv17bxswaprefect/venv/lib
no disk space remaining
Zanie03/27/2023, 3:35 PM
jpuris03/27/2023, 3:36 PM
I think we explicitly follow symlinksHi @Zanie, does that mean this is not intended behaviour? I can produce a MRE, if necessary. To my knowledge, the lib and lib64 symlink exists only for Debian bases distributions 🤷
Zanie03/27/2023, 3:37 PM
If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree and the metadata of the original links will be copied as far as the platform allows; if false or omitted, the contents and metadata of the linked files are copied to the new tree.
When symlinks is false, if the file pointed by the symlink doesn’t exist, an exception will be added in the list of errors raised in anexception at the end of the copy process. You can set the optional _ignore_dangling_symlinks_ flag to true if you want to silence this exception. Notice that this option has no effect on platforms that don’t support
jpuris03/27/2023, 3:40 PM
Zanie03/27/2023, 3:41 PM