https://prefect.io logo
Join Slack
Powered by
I am encountering `Permission denied` errors with ...
# ask-community
t
I am encountering 
Permission denied
errors with the github storage blocks since today. Has anyone seen this as well? The error appears if the github repo is already present from a previous run. If the repo is deleted the gitrepo is cloned without any error.
✅ 1
Copy code
shutil.Error: [('/tmp/tmpzw4y7qtmprefect/.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', "[Errno 13] Permission denied: './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', "[Errno 13] Permission denied: './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', "[Errno 13] Permission denied: './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', "[Errno 13] Permission denied: './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', "[Errno 13] Permission denied: './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', "[Errno 13] Permission denied: './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', "[Errno 13] Permission denied: './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/9d/10168d3633499e25522a57c8109767f0819735', './.git/objects/9d/10168d3633499e25522a57c8109767f0819735', "[Errno 13] Permission denied: './.git/objects/9d/10168d3633499e25522a57c8109767f0819735'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', "[Errno 13] Permission denied: './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', "[Errno 13] Permission denied: './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', "[Errno 13] Permission denied: './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/f9/53d1d075afbc06216855d7855503f9ea9b70a0', './.git/objects/f9/53d1d075afbc06216855d7855503f9ea9b70a0', "[Errno 13] Permission denied: './.git/objects/f9/53d1d075afbc06216855d7855503f9ea9b70a0'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', "[Errno 13] Permission denied: './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', "[Errno 13] Permission denied: './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', "[Errno 13] Permission denied: './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/be/e211a193292471d39e45698d228ee54039d802', './.git/objects/be/e211a193292471d39e45698d228ee54039d802', "[Errno 13] Permission denied: './.git/objects/be/e211a193292471d39e45698d228ee54039d802'")]
a
Can you rotate your personal access token?
t
Exchanged the token, but this did not solve the issue. The issue seems to be on the filesystem level, but I don't understand how cloning the first time works well but the second time fails.
Some of the objects in 
prefect-working-dir/.git/objects
have only read only permissions.
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/filesystems.py", line 917, in get_directory
    copytree(src=content_source, dst=content_destination, dirs_exist_ok=True)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/shutil.py", line 568, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/shutil.py", line 522, in _copytree
    raise Error(errors)
shutil.Error: [('/tmp/tmpjf2bug7lprefect/.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', "[Errno 13] Permission denied: './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', "[Errno 13] Permission denied: './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', "[Errno 13] Permission denied: './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', "[Errno 13] Permission denied: './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', "[Errno 13] Permission denied: './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', "[Errno 13] Permission denied: './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', "[Errno 13] Permission denied: './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/9d/10168d3633499e25522a57c8109767f0819735', './.git/objects/9d/10168d3633499e25522a57c8109767f0819735', "[Errno 13] Permission denied: './.git/objects/9d/10168d3633499e25522a57c8109767f0819735'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', "[Errno 13] Permission denied: './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', "[Errno 13] Permission denied: './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', "[Errno 13] Permission denied: './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/fc/09abcf4631f147623fcd3f0f1cd9f9af36cb7c', './.git/objects/fc/09abcf4631f147623fcd3f0f1cd9f9af36cb7c', "[Errno 13] Permission denied: './.git/objects/fc/09abcf4631f147623fcd3f0f1cd9f9af36cb7c'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/a8/c3a0a8b0be94c400fde72a6b045c8759c5167f', './.git/objects/a8/c3a0a8b0be94c400fde72a6b045c8759c5167f', "[Errno 13] Permission denied: './.git/objects/a8/c3a0a8b0be94c400fde72a6b045c8759c5167f'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', "[Errno 13] Permission denied: './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', "[Errno 13] Permission denied: './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', "[Errno 13] Permission denied: './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/be/e211a193292471d39e45698d228ee54039d802', './.git/objects/be/e211a193292471d39e45698d228ee54039d802', "[Errno 13] Permission denied: './.git/objects/be/e211a193292471d39e45698d228ee54039d802'")]
Alright, the issue was that the permissions of the working directory were messed up.
Hm. Still not good. Somehow the working directory permissions get messed up while the process infrastructure is setup. When I create the directory the first time I sets permissions to 
rwxrwxr-x
. Then I run the flow and the permissions change to 
rwx------
. The next run fails due to permission denied on copying the git-repo contents from tmp to the working dir.
I can work around it by setting chmod 775 -R on the working dir in the infrastructure commands. Should I file an issue?
a
It looks like you are running this in a very restricted environment, correct? We can certainly add that info to the docs but not sure what we can do about it otherwise. Code is cloned into a temp directory for execution Thanks for the update, nice troubleshooting!
t
True, my environment is pretty restricted. Out of curiosity, why is the code cloned into a temp dir and then copied to the working dir, instead of directly cloned into the working dir?
p
What python version are you running? Is it 3.7 by chance? And what prefect version are you using?
t
python 3.9.13 and prefect 2.6.9
p
We changed from the 
distutils
implementation of 
copy_tree
to 
shutil.copytree
because 
distutils
is scheduled to be deprecated. I'm wondering if this caused a regression. https://github.com/PrefectHQ/prefect/pull/7441
t
Maybe. I observe this behavior since yesterday. I also upgraded yesterday to prefect 2.6.9.
On Monday I used prefect main branch and did not encounter this.
Happy to test this if you let me know which commits I should install for testing.
p
Can you go into your prefect installation, under 
src/prefect/filesystems.py
and delete this import: 
from prefect.utilities.compat import copytree
and add this import: 
from distutils.dir_util import copy_tree as copytree
?
and then see if it works
t
Different signature:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/filesystems.py", line 917, in get_directory
    copytree(src=content_source, dst=content_destination, dirs_exist_ok=True)
TypeError: copy_tree() got an unexpected keyword argument 'dirs_exist_ok'
p
Ah shoot - that's right. If you're in an experimental mood can you go to line 
917
and change the signature to 
copytree(src=content_source, dst=content_destination)
and then try again?
gratitude thank you 1
t
Yes! This fixes the issue.
Reverted back to 2.6.9 and the error is back as well.
Thanks for looking into this with me. I have to run now, workday is over.
👍 2
Is there a github issue I could follow or is this not considered a bug?
p
I don't think there's an issue for this. I'm just getting back from the Thanksgiving break, so my to-do list is packed this morning. Would you be able make one when you get a moment? I'll discuss it with the team this week and see if it we would consider this to be a bug or expected behavior.
t
Welcome back, hope it was great. I will open an issue.
🙏 1
9 Views
Open in Slack
PreviousNext
Powered by