Jens Freund
10/27/2022, 11:01 PMFile "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:16:29.384 | INFO | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.
To solve the PermissionError, I tried two approaches, which unfortunately both failed:
1. Try to change the PREFECT_LOCAL_STORAGE_PATH
2. Running Powershell, from which I start Prefect, as admin
My problem seems to be practically the same as that reported by @Mike He in the discourse forum (https://discourse.prefect.io/t/agent-debugging-agent-execution-error-with-ftp-storage-in-windows/1587) and here in Slack (https://prefect-community.slack.com/archives/C03D12VV4NN/p1663319368100279). (The only difference I see is that in my case, the last error of the traceback comes from the paramiko library instead of fsspec.) I saw that he already developed a workaround, which however includes changing the Prefect source code which I have not tried yet.
I would write details, e.g. the full traceback, in the thread. If anyone had an idea or a hint how this problem could be solved, preferably without manually changing the Prefect source code, you would help me again a lot. Thank you very much in advance!$ prefect agent start -q 'ftp-test'
PS C:\Users\Testbenutzer\prefect_test_project> prefect agent start -q 'ftp-test'
Starting v2.6.5 agent with ephemeral API...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): ftp-test...
23:16:26.317 | INFO | prefect.agent - Submitting flow run '58f7405d-fe15-4f1b-b4ea-9d66183ec8f3'
23:16:26.408 | INFO | prefect.infrastructure.process - Opening process 'upbeat-crab'...
23:16:26.412 | INFO | prefect.agent - Completed submission of flow run '58f7405d-fe15-4f1b-b4ea-9d66183ec8f3'
23:16:28.966 | ERROR | Flow run 'upbeat-crab' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\filesystems.py", line 310, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\fsspec\implementations\sftp.py", line 134, in get
self.ftp.get(rpath, lpath)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:16:29.384 | INFO | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.
Screenshot of Prefect settings:PS C:\Users\Testbenutzer\prefect_test_project> prefect config set PREFECT_LOCAL_STORAGE_PATH='C:\Users\Testbenutzer\prefect-storage'
Set 'PREFECT_LOCAL_STORAGE_PATH' to 'C:\\Users\\Testbenutzer\\prefect-storage'.
Updated profile 'default'.
After a restart of the orion server, the new path "C:\Users\Testbenutzer\prefect-storage"
is correctly shown in the settings (see screenshot).
However, when starting the flow run, the error persists, again with "Permission denied: '.'"
instead of the custom path "C:\Users\Testbenutzer\prefect-storage"
from the settings window:
PS C:\Users\Testbenutzer\prefect_test_project> prefect agent start -q 'ftp-test'
Starting v2.6.5 agent with ephemeral API...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): ftp-test...
23:24:11.036 | INFO | prefect.agent - Submitting flow run '496f43f9-bdcc-4652-9244-8ab5d5c5176c'
23:24:11.127 | INFO | prefect.infrastructure.process - Opening process 'rare-manatee'...
23:24:11.132 | INFO | prefect.agent - Completed submission of flow run '496f43f9-bdcc-4652-9244-8ab5d5c5176c'
23:24:13.984 | ERROR | Flow run 'rare-manatee' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\filesystems.py", line 310, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\fsspec\implementations\sftp.py", line 134, in get
self.ftp.get(rpath, lpath)
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:24:14.457 | INFO | prefect.infrastructure.process - Process 'rare-manatee' exited cleanly.
Q
10/28/2022, 12:32 AMprefect.filesystems.RemotefileSystem
uses fsspec
under the hood.
The assumption is that fsspec
filesystems can be used in the same fashion without having to know the internals of any implementation.
get
method for most fsspec
fs implementations is able to copy directories.
fsspec.implementations.sftp.SFTPFileSystem.get
can't copy directories (I'm guessing that's because paramiko.SFTPClient
can't).
I don't really know how one can easily fix this. Maybe creating an issue at fsspec could be a decent start?Jens Freund
10/28/2022, 10:49 AM