<@ULVA73B9P>, we switched to using Github as our f...
# ask-marvin
d
@Marvin, we switched to using Github as our filesystem recently and now we intermittently see the following error, even though github claims there were no incidents in the relevant time period. Is there anything we can do to prevent this or be more resilient to it etc?
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  ....
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 995, in get_directory
    raise OSError(f"Failed to pull from remote:\n {err_stream.read()}")
OSError: Failed to pull from remote:
 Cloning into '/tmp/tmpwjpgmu6mprefect'...
fatal: unable to access '<https://github.com/MY_REPO_NAME_HERE>': gnutls_handshake() failed: The TLS connection was non-properly terminated.
m
Your message was too long by 18 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
, we switched to using Github as our filesystem recently and now we intermittently see the following error, even though github claims there were no incidents in the relevant time period. Is there anything we can do to prevent this or be more resilient to it etc?
```Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 424, in retrieve_flow_then_begin_flow_run else await load_flow_from_flow_run(flow_run, client=client) File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 252, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 995, in get_directory raise OSError(f"Failed to pull from remote:\n {err_stream.read()}") OSError: Failed to pull from remote: Cloning into '/tmp/tmpwjpgmu6mprefect'... fatal: unable to access 'https://github.com/MY_REPO_NAME_HERE': g
d
it's shorter now
@Marvin
m
It seems like you are experiencing intermittent TLS handshake issues when your Prefect flows try to retrieve something from GitHub. This error could be related to network conditions, GitHub's API behavior, or configuration in Prefect's interaction with the GitHub filesystem. However, based on the returned results, there isn’t a direct solution to the TLS handshake error in the context of Prefect documented in their release notes, documentation, or reported GitHub issues. The only somewhat related items are about
RemoteFileSystem
issues, but they don't match your error directly. As for general advice to increase resiliency: 1. Retry Logic: Implementing retry logic with exponential backoff could help to overcome temporary network issues or intermittent GitHub API glitches. 2. Monitoring and Alerting: Tools like Sentry or Prometheus can help to monitor your system and alert you to these errors as they happen, allowing quicker response. 3. Alternative Storage: If the issue persists, consider using an alternative to GitHub as your filesystem for critical workflows, such as cloud storage options like Amazon S3 or Azure Blob Storage that might offer more reliability for your use case. 4. Network Troubleshooting: Checking your network configuration might help. Ensure that there are no firewalls or proxies interfering with the connections. If you need to further investigate this issue or the general error persists, I recommend reaching out to Prefect's support or filing an issue on their GitHub repository with detailed information about your setup and the error you are encountering. Remember, while GitHub's status page might not show incidents, this doesn't rule out brief or localized issues that could affect your workflows.
d
thanks marvin, i would be interested to hear if any humans have any additional thoughts
actually maybe it was a proper incident in github - https://www.githubstatus.com/incidents/fr6fgvdn41sw