I am encountering `Permission denied` errors with the github Prefect Community #prefect-community

I am encountering `Permission denied` errors with ...

Tim-Oliver

11/23/2022, 4:59 PM

I am encountering

Permission denied

errors with the github storage blocks since today. Has anyone seen this as well? The error appears if the github repo is already present from a previous run. If the repo is deleted the gitrepo is cloned without any error.

✅ 1

Tim-Oliver

11/23/2022, 4:59 PM

Copy code

shutil.Error: [('/tmp/tmpzw4y7qtmprefect/.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', "[Errno 13] Permission denied: './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', "[Errno 13] Permission denied: './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', "[Errno 13] Permission denied: './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', "[Errno 13] Permission denied: './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', "[Errno 13] Permission denied: './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', "[Errno 13] Permission denied: './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', "[Errno 13] Permission denied: './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/9d/10168d3633499e25522a57c8109767f0819735', './.git/objects/9d/10168d3633499e25522a57c8109767f0819735', "[Errno 13] Permission denied: './.git/objects/9d/10168d3633499e25522a57c8109767f0819735'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', "[Errno 13] Permission denied: './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', "[Errno 13] Permission denied: './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', "[Errno 13] Permission denied: './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/f9/53d1d075afbc06216855d7855503f9ea9b70a0', './.git/objects/f9/53d1d075afbc06216855d7855503f9ea9b70a0', "[Errno 13] Permission denied: './.git/objects/f9/53d1d075afbc06216855d7855503f9ea9b70a0'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', "[Errno 13] Permission denied: './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', "[Errno 13] Permission denied: './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', "[Errno 13] Permission denied: './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6'"), ('/tmp/tmpzw4y7qtmprefect/.git/objects/be/e211a193292471d39e45698d228ee54039d802', './.git/objects/be/e211a193292471d39e45698d228ee54039d802', "[Errno 13] Permission denied: './.git/objects/be/e211a193292471d39e45698d228ee54039d802'")]

Anna Geller

11/23/2022, 8:00 PM

Can you rotate your personal access token?

Anna Geller

11/23/2022, 8:01 PM

here is how https://medium.com/the-prefect-blog/modular-data-stack-build-a-data-platform-with-prefect-dbt-and-snowflake-part-6-7e6d69296649#c31b

Tim-Oliver

11/24/2022, 7:29 AM

Exchanged the token, but this did not solve the issue. The issue seems to be on the filesystem level, but I don't understand how cloning the first time works well but the second time fails.

Tim-Oliver

11/24/2022, 7:33 AM

Some of the objects in

prefect-working-dir/.git/objects

have only read only permissions.

Tim-Oliver

11/24/2022, 8:10 AM

Copy code

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/filesystems.py", line 917, in get_directory
    copytree(src=content_source, dst=content_destination, dirs_exist_ok=True)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/shutil.py", line 568, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/shutil.py", line 522, in _copytree
    raise Error(errors)
shutil.Error: [('/tmp/tmpjf2bug7lprefect/.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f', "[Errno 13] Permission denied: './.git/objects/51/16fdcab0c85837d53ac9ee05bbeb7b0831429f'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97', "[Errno 13] Permission denied: './.git/objects/b7/6ed055126028e448438c6c1503991c8a70de97'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786', "[Errno 13] Permission denied: './.git/objects/d5/222fd9a29067934daed5b4b0fb6307f70cf786'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97', "[Errno 13] Permission denied: './.git/objects/f4/0a470220a9ccc37f0005f7978e8df1695dfb97'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391', "[Errno 13] Permission denied: './.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261', "[Errno 13] Permission denied: './.git/objects/35/9b656038f5c35061d26f5f73cdd258d29d0261'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529', "[Errno 13] Permission denied: './.git/objects/15/9200a08fdc5741d070d432338b9c2ef18ed529'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/9d/10168d3633499e25522a57c8109767f0819735', './.git/objects/9d/10168d3633499e25522a57c8109767f0819735', "[Errno 13] Permission denied: './.git/objects/9d/10168d3633499e25522a57c8109767f0819735'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522', "[Errno 13] Permission denied: './.git/objects/3e/5551707d7888df29e5182461fcc324efeda522'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634', "[Errno 13] Permission denied: './.git/objects/b2/c164d3a81663f1823a1baebc1b98f589fc8634'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da', "[Errno 13] Permission denied: './.git/objects/62/dc4b9d007c19610d4c8cf9b43b7851c2b315da'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/fc/09abcf4631f147623fcd3f0f1cd9f9af36cb7c', './.git/objects/fc/09abcf4631f147623fcd3f0f1cd9f9af36cb7c', "[Errno 13] Permission denied: './.git/objects/fc/09abcf4631f147623fcd3f0f1cd9f9af36cb7c'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/a8/c3a0a8b0be94c400fde72a6b045c8759c5167f', './.git/objects/a8/c3a0a8b0be94c400fde72a6b045c8759c5167f', "[Errno 13] Permission denied: './.git/objects/a8/c3a0a8b0be94c400fde72a6b045c8759c5167f'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da', "[Errno 13] Permission denied: './.git/objects/3a/d797acf2f668e46454014df230ce4a5a76b1da'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42', "[Errno 13] Permission denied: './.git/objects/79/c899a064ac6800264c61d9cfb04104df966a42'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6', "[Errno 13] Permission denied: './.git/objects/cf/b8a3d7361b654ac455084b181cadd7a5f3c4e6'"), ('/tmp/tmpjf2bug7lprefect/.git/objects/be/e211a193292471d39e45698d228ee54039d802', './.git/objects/be/e211a193292471d39e45698d228ee54039d802', "[Errno 13] Permission denied: './.git/objects/be/e211a193292471d39e45698d228ee54039d802'")]

Tim-Oliver

11/24/2022, 1:08 PM

Alright, the issue was that the permissions of the working directory were messed up.

Tim-Oliver

11/24/2022, 2:28 PM

Hm. Still not good. Somehow the working directory permissions get messed up while the process infrastructure is setup. When I create the directory the first time I sets permissions to

rwxrwxr-x

. Then I run the flow and the permissions change to

rwx------

. The next run fails due to permission denied on copying the git-repo contents from tmp to the working dir.

Tim-Oliver

11/24/2022, 2:52 PM

I can work around it by setting chmod 775 -R on the working dir in the infrastructure commands. Should I file an issue?

Anna Geller

11/24/2022, 3:04 PM

It looks like you are running this in a very restricted environment, correct? We can certainly add that info to the docs but not sure what we can do about it otherwise. Code is cloned into a temp directory for execution Thanks for the update, nice troubleshooting!

Tim-Oliver

11/24/2022, 3:09 PM

True, my environment is pretty restricted. Out of curiosity, why is the code cloned into a temp dir and then copied to the working dir, instead of directly cloned into the working dir?

Peyton Runyan

11/24/2022, 3:34 PM

What python version are you running? Is it 3.7 by chance? And what prefect version are you using?

Tim-Oliver

11/24/2022, 3:39 PM

python 3.9.13 and prefect 2.6.9

Peyton Runyan

11/24/2022, 3:48 PM

We changed from the

distutils

implementation of

copy_tree

shutil.copytree

because

distutils

is scheduled to be deprecated. I'm wondering if this caused a regression. https://github.com/PrefectHQ/prefect/pull/7441

Tim-Oliver

11/24/2022, 3:51 PM

Maybe. I observe this behavior since yesterday. I also upgraded yesterday to prefect 2.6.9.

Tim-Oliver

11/24/2022, 3:52 PM

On Monday I used prefect main branch and did not encounter this.

Tim-Oliver

11/24/2022, 3:52 PM

Happy to test this if you let me know which commits I should install for testing.

Peyton Runyan

11/24/2022, 3:53 PM

Can you go into your prefect installation, under

src/prefect/filesystems.py

and delete this import:

from prefect.utilities.compat import copytree

and add this import:

from distutils.dir_util import copy_tree as copytree

Peyton Runyan

11/24/2022, 3:53 PM

and then see if it works

Tim-Oliver

11/24/2022, 4:10 PM

Different signature:

Copy code

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/tungstenfs/scratch/gmicro_share/_prefect/miniconda3/envs/test_gfriedri-em-alignment-flows/lib/python3.9/site-packages/prefect/filesystems.py", line 917, in get_directory
    copytree(src=content_source, dst=content_destination, dirs_exist_ok=True)
TypeError: copy_tree() got an unexpected keyword argument 'dirs_exist_ok'

Peyton Runyan

11/24/2022, 4:17 PM

Ah shoot - that's right. If you're in an experimental mood can you go to line

and change the signature to

copytree(src=content_source, dst=content_destination)

and then try again?

gratitude thank you 1

Tim-Oliver

11/24/2022, 4:22 PM

Yes! This fixes the issue.

Tim-Oliver

11/24/2022, 4:22 PM

Reverted back to 2.6.9 and the error is back as well.

Tim-Oliver

11/24/2022, 4:26 PM

Thanks for looking into this with me. I have to run now, workday is over.

👍 2

Tim-Oliver

11/28/2022, 8:16 AM

Is there a github issue I could follow or is this not considered a bug?

Peyton Runyan

11/28/2022, 1:05 PM

I don't think there's an issue for this. I'm just getting back from the Thanksgiving break, so my to-do list is packed this morning. Would you be able make one when you get a moment? I'll discuss it with the team this week and see if it we would consider this to be a bug or expected behavior.

Tim-Oliver

11/28/2022, 1:10 PM

Welcome back, hope it was great. I will open an issue.

🙏 1

Tim-Oliver

11/28/2022, 1:31 PM

https://github.com/PrefectHQ/prefect/issues/7664

🔥 1

4 Views

Open in Slack

Previous Next