https://prefect.io logo
Channels
Powered by
j

Jens Freund

10/27/2022, 11:01 PM
Hello all, After my last problem was immediately and perfectly solved here in the channel (thank you, again, @Q !), I would like to ask if anyone could help me with a second question, concerning the same example project (https://prefect-community.slack.com/archives/CL09KU1K7/p1666555308227089)? Namely, now that the creation of the deployment on the SFTP remote storage block works, I'm unfortunately unable to start a flow run from it. Specifically, I started a local Prefect agent and afterwards a local flow run from the UI (Run -> Now with defaults). After that, the paramiko library gives the following error:
Copy code
File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
    with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:16:29.384 | INFO    | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.
To solve the PermissionError, I tried two approaches, which unfortunately both failed: 1. Try to change the 
PREFECT_LOCAL_STORAGE_PATH
2. Running Powershell, from which I start Prefect, as admin My problem seems to be practically the same as that reported by @Mike He in the discourse forum (https://discourse.prefect.io/t/agent-debugging-agent-execution-error-with-ftp-storage-in-windows/1587) and here in Slack (https://prefect-community.slack.com/archives/C03D12VV4NN/p1663319368100279). (The only difference I see is that in my case, the last error of the traceback comes from the paramiko library instead of fsspec.) I saw that he already developed a workaround, which however includes changing the Prefect source code which I have not tried yet. I would write details, e.g. the full traceback, in the thread. If anyone had an idea or a hint how this problem could be solved, preferably without manually changing the Prefect source code, you would help me again a lot. Thank you very much in advance!
✅ 1
Flow run error with default settings:
Copy code
$ prefect agent start -q 'ftp-test'
PS C:\Users\Testbenutzer\prefect_test_project> prefect agent start -q 'ftp-test'
Starting v2.6.5 agent with ephemeral API...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): ftp-test...
23:16:26.317 | INFO    | prefect.agent - Submitting flow run '58f7405d-fe15-4f1b-b4ea-9d66183ec8f3'
23:16:26.408 | INFO    | prefect.infrastructure.process - Opening process 'upbeat-crab'...
23:16:26.412 | INFO    | prefect.agent - Completed submission of flow run '58f7405d-fe15-4f1b-b4ea-9d66183ec8f3'
23:16:28.966 | ERROR   | Flow run 'upbeat-crab' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\filesystems.py", line 310, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\fsspec\implementations\sftp.py", line 134, in get
    self.ftp.get(rpath, lpath)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
    with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:16:29.384 | INFO    | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.
Screenshot of Prefect settings:
Command to change the local storage path:
Copy code
PS C:\Users\Testbenutzer\prefect_test_project> prefect config set PREFECT_LOCAL_STORAGE_PATH='C:\Users\Testbenutzer\prefect-storage'
Set 'PREFECT_LOCAL_STORAGE_PATH' to 'C:\\Users\\Testbenutzer\\prefect-storage'.
Updated profile 'default'.
After a restart of the orion server, the new path 
"C:\Users\Testbenutzer\prefect-storage"
is correctly shown in the settings (see screenshot). However, when starting the flow run, the error persists, again with 
"Permission denied: '.'"
instead of the custom path 
"C:\Users\Testbenutzer\prefect-storage"
from the settings window:
Copy code
PS C:\Users\Testbenutzer\prefect_test_project> prefect agent start -q 'ftp-test' 
Starting v2.6.5 agent with ephemeral API...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____ 
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |  
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|  


Agent started! Looking for work from queue(s): ftp-test... 
23:24:11.036 | INFO    | prefect.agent - Submitting flow run '496f43f9-bdcc-4652-9244-8ab5d5c5176c'
23:24:11.127 | INFO    | prefect.infrastructure.process - Opening process 'rare-manatee'...
23:24:11.132 | INFO    | prefect.agent - Completed submission of flow run '496f43f9-bdcc-4652-9244-8ab5d5c5176c'
23:24:13.984 | ERROR   | Flow run 'rare-manatee' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\prefect\filesystems.py", line 310, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\fsspec\implementations\sftp.py", line 134, in get
    self.ftp.get(rpath, lpath)
  File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
    with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:24:14.457 | INFO    | prefect.infrastructure.process - Process 'rare-manatee' exited cleanly.
q

Q

10/28/2022, 12:32 AM
I think the problem is on `fsspec`'s side here. 
prefect.filesystems.RemotefileSystem
uses 
fsspec
under the hood. The assumption is that 
fsspec
filesystems can be used in the same fashion without having to know the internals of any implementation. 
get
method for most 
fsspec
fs implementations is able to copy directories. 
fsspec.implementations.sftp.SFTPFileSystem.get
can't copy directories (I'm guessing that's because 
paramiko.SFTPClient
can't). I don't really know how one can easily fix this. Maybe creating an issue at fsspec could be a decent start?
🙏 2
🙌 1
j

Jens Freund

10/28/2022, 10:49 AM
Thank you very, very much again for your support @Q and the very understandable error description! You helped me again a lot! I will create an issue at fsspec and link it here in the chat.
Powered by