Daniel Manson
03/11/2024, 12:07 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
....
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 995, in get_directory
raise OSError(f"Failed to pull from remote:\n {err_stream.read()}")
OSError: Failed to pull from remote:
Cloning into '/tmp/tmpwjpgmu6mprefect'...
fatal: unable to access '<https://github.com/MY_REPO_NAME_HERE>': gnutls_handshake() failed: The TLS connection was non-properly terminated.
Marvin
03/11/2024, 12:07 PM, we switched to using Github as our filesystem recently and now we intermittently see the following error, even though github claims there were no incidents in the relevant time period. Is there anything we can do to prevent this or be more resilient to it etc?```Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 424, in retrieve_flow_then_begin_flow_run else await load_flow_from_flow_run(flow_run, client=client) File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 252, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 995, in get_directory raise OSError(f"Failed to pull from remote:\n {err_stream.read()}") OSError: Failed to pull from remote: Cloning into '/tmp/tmpwjpgmu6mprefect'... fatal: unable to access 'https://github.com/MY_REPO_NAME_HERE': g
Daniel Manson
03/11/2024, 12:08 PMDaniel Manson
03/11/2024, 12:08 PMMarvin
03/11/2024, 12:09 PMRemoteFileSystem
issues, but they don't match your error directly.
As for general advice to increase resiliency:
1. Retry Logic: Implementing retry logic with exponential backoff could help to overcome temporary network issues or intermittent GitHub API glitches.
2. Monitoring and Alerting: Tools like Sentry or Prometheus can help to monitor your system and alert you to these errors as they happen, allowing quicker response.
3. Alternative Storage: If the issue persists, consider using an alternative to GitHub as your filesystem for critical workflows, such as cloud storage options like Amazon S3 or Azure Blob Storage that might offer more reliability for your use case.
4. Network Troubleshooting: Checking your network configuration might help. Ensure that there are no firewalls or proxies interfering with the connections.
If you need to further investigate this issue or the general error persists, I recommend reaching out to Prefect's support or filing an issue on their GitHub repository with detailed information about your setup and the error you are encountering.
Remember, while GitHub's status page might not show incidents, this doesn't rule out brief or localized issues that could affect your workflows.Daniel Manson
03/11/2024, 12:22 PMDaniel Manson
03/12/2024, 9:58 AM